[==========] Running 23 tests from 3 test cases. [----------] Global test environment set-up. [----------] 9 tests from SecurityITest [ RUN ] SecurityITest.SmokeTestAsAuthorizedUser Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): setting up network... May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): skipping unrecognized local address family 17 May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): skipping unrecognized local address family 17 May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): set up 1 sockets May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): commencing operation krb5kdc: starting... W0506 07:56:32.681979 785 mini_kdc.cc:117] Time spent starting KDC: real 0.190s user 0.005s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:32.910228 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:50742 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:33.191182 1027 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:33.191716 1027 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:33.192191 1027 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:33.209058 1027 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:33.209272 1027 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:33.209460 1027 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:33.209643 1027 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:33.209877 1027 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:33.210105 1027 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:33.248033 1027 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:50742 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:33.268848 1027 master_main.cc:73] Initializing master server... I0506 07:56:33.272835 1027 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 192178us May 06 07:56:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:33.280258 1027 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:33.293583 1027 fs_manager.cc:260] Metadata directory not provided I0506 07:56:33.293839 1027 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:33.294173 1027 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:33.294394 1027 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:33.319893 1027 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "0bf0e56042e94995852891e9b599f500" format_stamp: "Formatted at 2018-05-06 14:56:33 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:33.320992 1027 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "0bf0e56042e94995852891e9b599f500" format_stamp: "Formatted at 2018-05-06 14:56:33 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:33.338570 1027 fs_manager.cc:495] Time spent creating directory manager: real 0.017s user 0.011s sys 0.004s I0506 07:56:33.340340 1027 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:33.341811 1027 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:33.360292 1027 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.000s sys 0.005s I0506 07:56:33.360532 1027 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/wal uuid: "0bf0e56042e94995852891e9b599f500" format_stamp: "Formatted at 2018-05-06 14:56:33 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:33.360945 1027 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:33.621896 1027 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:33.625440 1027 master_main.cc:76] Starting Master server... I0506 07:56:33.739126 1027 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:50742 I0506 07:56:33.740092 1027 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:56:33.740192 1027 webserver.cc:184] Document root disabled I0506 07:56:33.742399 1027 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:34565/ I0506 07:56:33.743181 1027 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:33.747581 1275 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:33.748497 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 1027 WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:56:33.766019 1275 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Bootstrap starting. I0506 07:56:33.769527 1275 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: No blocks or log segments found. Creating new log. I0506 07:56:33.771533 1275 log.cc:520] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Log is configured to *not* fsync() on all Append() calls I0506 07:56:33.775212 1275 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:56:33.779553 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:33.793380 1275 raft_consensus.cc:303] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } I0506 07:56:33.793956 1275 raft_consensus.cc:329] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:33.794198 1275 raft_consensus.cc:604] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf0e56042e94995852891e9b599f500, State: Initialized, Role: FOLLOWER I0506 07:56:33.794916 1275 consensus_queue.cc:227] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } I0506 07:56:33.795534 1275 raft_consensus.cc:343] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:33.795828 1275 raft_consensus.cc:435] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:33.796090 1275 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Snoozing failure detection for 1.542s (starting election) I0506 07:56:33.796355 1275 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:33.800701 1275 raft_consensus.cc:457] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } I0506 07:56:33.801336 1275 leader_election.cc:257] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:33.805235 1301 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Snoozing failure detection for 1.606s (election complete) I0506 07:56:33.805521 1301 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:33.813367 1301 raft_consensus.cc:571] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf0e56042e94995852891e9b599f500, State: Running, Role: LEADER I0506 07:56:33.813879 1301 consensus_queue.cc:204] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } I0506 07:56:33.818936 1303 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf0e56042e94995852891e9b599f500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } } I0506 07:56:33.819378 1303 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: This master's current role is: LEADER I0506 07:56:33.822181 1304 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf0e56042e94995852891e9b599f500. Latest consensus state: current_term: 1 leader_uuid: "0bf0e56042e94995852891e9b599f500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } } I0506 07:56:33.928788 1304 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: This master's current role is: LEADER I0506 07:56:33.820710 1275 sys_catalog.cc:335] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0bf0e56042e94995852891e9b599f500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf0e56042e94995852891e9b599f500" member_type: VOTER } } I0506 07:56:33.936414 1275 sys_catalog.cc:338] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: This master's current role is: LEADER I0506 07:56:33.937826 1275 sys_catalog.cc:422] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:33.945402 1355 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:56:33.948506 1027 master_main.cc:79] Master server successfully started. I0506 07:56:33.946111 1311 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:56:33.968071 1311 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:34.044945 1283 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:34.045490 1283 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:56:34.046066 1283 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:34.065369 1283 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:34.065516 1283 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:34.065830 1283 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:56:34.066366 1283 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:56:34.102612 1283 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:34.112666 1283 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:34.115644 1283 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 192678us I0506 07:56:34.118831 1311 catalog_manager.cc:888] Generated new certificate authority record May 06 07:56:34 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:34.121261 1283 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:56:34.121394 1311 catalog_manager.cc:1010] Loading token signing keys... I0506 07:56:34.126680 1283 fs_manager.cc:260] Metadata directory not provided I0506 07:56:34.126830 1283 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:34.127053 1283 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:56:34.127190 1283 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:34.137647 1283 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "9bc96c3eb634481883f01cbea31443b2" format_stamp: "Formatted at 2018-05-06 14:56:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:34.138383 1283 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "9bc96c3eb634481883f01cbea31443b2" format_stamp: "Formatted at 2018-05-06 14:56:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:34.149852 1311 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Generated new TSK 0 I0506 07:56:34.157282 1283 fs_manager.cc:495] Time spent creating directory manager: real 0.018s user 0.009s sys 0.006s I0506 07:56:34.158594 1283 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:34.159842 1283 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:34.169524 1283 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0506 07:56:34.169672 1283 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/wal uuid: "9bc96c3eb634481883f01cbea31443b2" format_stamp: "Formatted at 2018-05-06 14:56:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:34.169919 1283 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:34.378866 1283 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:34.382056 1283 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:34.385437 1283 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:35.075614 1283 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:46226 I0506 07:56:35.076985 1283 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:56:35.077184 1283 webserver.cc:184] Document root disabled I0506 07:56:35.085597 1283 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:60187/ I0506 07:56:35.086688 1283 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:56:35.095664 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 1283 I0506 07:56:35.123214 1283 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. May 06 07:56:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:56:35.161824 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:35.176981 1586 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:35.097430 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50742 0506 07:56:35.116725 (+ 19295us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:35.116916 (+ 191us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:35.140449 (+ 23533us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:35.142483 (+ 2034us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:35.142513 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:35.143291 (+ 778us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:35.143548 (+ 257us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:35.143963 (+ 415us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:35.144005 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:35.145389 (+ 1384us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:35.145413 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:35.148381 (+ 2968us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:35.148421 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:35.151597 (+ 3176us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:35.151647 (+ 50us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:35.152088 (+ 441us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:35.152358 (+ 270us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:35.152377 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:35.156401 (+ 4024us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:35.166934 (+ 10533us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:35.166964 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:35.166988 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:35.168206 (+ 1218us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:35.170287 (+ 2081us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:35.170312 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:35.170330 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:35.170533 (+ 203us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:35.172608 (+ 2075us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:35.172661 (+ 53us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:35.173072 (+ 411us) client_negotiation.cc:695] Sending connection context 0506 07:56:35.173553 (+ 481us) client_negotiation.cc:225] Negotiation successful 0506 07:56:35.173943 (+ 390us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13759,"spinlock_wait_cycles":24448,"thread_start_us":12968,"threads_started":1} I0506 07:56:35.180559 1587 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:35.098967 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:39934 0506 07:56:35.107559 (+ 8592us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:35.107600 (+ 41us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:35.117700 (+ 10100us) server_negotiation.cc:373] Connection header received 0506 07:56:35.141226 (+ 23526us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:35.141273 (+ 47us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:35.141535 (+ 262us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:35.141926 (+ 391us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:35.144216 (+ 2290us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:35.145092 (+ 876us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:35.148609 (+ 3517us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:35.150873 (+ 2264us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:35.151083 (+ 210us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:35.157309 (+ 6226us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:35.157384 (+ 75us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:35.157414 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:35.157491 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:35.166078 (+ 8587us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:35.168884 (+ 2806us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:35.168913 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:35.168937 (+ 24us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:35.169675 (+ 738us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:35.171186 (+ 1511us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:35.171214 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:35.171234 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:35.171854 (+ 620us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:35.172236 (+ 382us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:35.177973 (+ 5737us) server_negotiation.cc:264] Negotiation successful 0506 07:56:35.178387 (+ 414us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8229,"thread_start_us":10521,"threads_started":1} I0506 07:56:35.183506 1584 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50742 I0506 07:56:35.183830 1584 heartbeater.cc:426] Registering TS with master... I0506 07:56:35.184470 1584 heartbeater.cc:475] Master 127.0.196.126:50742 requested a full tablet report, sending... I0506 07:56:35.187201 1236 ts_manager.cc:81] Registered new tserver with Master: 9bc96c3eb634481883f01cbea31443b2 (127.0.196.65:46226) I0506 07:56:35.190152 1236 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:39934 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:35.441823 1600 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:35.442356 1600 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:56:35.442843 1600 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:35.460991 1600 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:35.461218 1600 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:35.461513 1600 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:56:35.462065 1600 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:56:35.520447 1600 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:35.536538 1600 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:35.538918 1600 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 193178us May 06 07:56:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:35.542855 1600 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:56:35.551422 1600 fs_manager.cc:260] Metadata directory not provided I0506 07:56:35.551641 1600 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:35.552026 1600 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:56:35.552279 1600 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:35.568260 1600 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "243df4b1bb1e4bc98a379649d4c98796" format_stamp: "Formatted at 2018-05-06 14:56:35 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:35.580448 1600 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "243df4b1bb1e4bc98a379649d4c98796" format_stamp: "Formatted at 2018-05-06 14:56:35 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:35.626893 1600 fs_manager.cc:495] Time spent creating directory manager: real 0.042s user 0.013s sys 0.014s I0506 07:56:35.628826 1600 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:35.629490 1600 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:35.649972 1600 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s I0506 07:56:35.650202 1600 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/wal uuid: "243df4b1bb1e4bc98a379649d4c98796" format_stamp: "Formatted at 2018-05-06 14:56:35 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:35.650598 1600 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:36.126893 1600 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:36.135800 1600 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:36.142802 1600 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:36.194344 1584 heartbeater.cc:467] Master 127.0.196.126:50742 was elected leader, sending a full tablet report... I0506 07:56:36.540385 1600 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:58246 I0506 07:56:36.542132 1600 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:56:36.542330 1600 webserver.cc:184] Document root disabled I0506 07:56:36.548363 1600 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:59051/ I0506 07:56:36.549474 1600 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:56:36.555723 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 1600 I0506 07:56:36.567317 1600 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. I0506 07:56:36.591410 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:56:36 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:36.607990 1806 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:36.553363 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:60591 0506 07:56:36.556816 (+ 3453us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:36.556844 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:36.558296 (+ 1452us) server_negotiation.cc:373] Connection header received 0506 07:56:36.582840 (+ 24544us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:36.582859 (+ 19us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:36.582966 (+ 107us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:36.583111 (+ 145us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:36.585692 (+ 2581us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:36.586103 (+ 411us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:36.588076 (+ 1973us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:36.589062 (+ 986us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:36.589225 (+ 163us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:36.593918 (+ 4693us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:36.593994 (+ 76us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:36.594020 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:36.594099 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:36.600437 (+ 6338us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:36.602399 (+ 1962us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:36.602417 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:36.602427 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:36.602861 (+ 434us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:36.603881 (+ 1020us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:36.603900 (+ 19us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:36.603910 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:36.604224 (+ 314us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:36.604561 (+ 337us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:36.605791 (+ 1230us) server_negotiation.cc:264] Negotiation successful 0506 07:56:36.606113 (+ 322us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3110,"thread_start_us":2773,"threads_started":1} I0506 07:56:36.610278 1807 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:36.554168 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50742 0506 07:56:36.558037 (+ 3869us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:36.558145 (+ 108us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:36.582465 (+ 24320us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:36.583366 (+ 901us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:36.583399 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:36.584321 (+ 922us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:36.584613 (+ 292us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:36.585454 (+ 841us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:36.585496 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:36.586280 (+ 784us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:36.586304 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:36.587877 (+ 1573us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:36.587916 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:36.589277 (+ 1361us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:36.589307 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:36.589709 (+ 402us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:36.589952 (+ 243us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:36.589964 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:36.593150 (+ 3186us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:36.600910 (+ 7760us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:36.600928 (+ 18us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:36.600942 (+ 14us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:36.601984 (+ 1042us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:36.603255 (+ 1271us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:36.603270 (+ 15us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:36.603281 (+ 11us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:36.603447 (+ 166us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:36.604691 (+ 1244us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:36.604717 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:36.605024 (+ 307us) client_negotiation.cc:695] Sending connection context 0506 07:56:36.605468 (+ 444us) client_negotiation.cc:225] Negotiation successful 0506 07:56:36.605916 (+ 448us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3190,"thread_start_us":2675,"threads_started":1} I0506 07:56:36.613162 1804 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50742 I0506 07:56:36.613495 1804 heartbeater.cc:426] Registering TS with master... I0506 07:56:36.614209 1804 heartbeater.cc:475] Master 127.0.196.126:50742 requested a full tablet report, sending... I0506 07:56:36.616272 1236 ts_manager.cc:81] Registered new tserver with Master: 243df4b1bb1e4bc98a379649d4c98796 (127.0.196.66:58246) I0506 07:56:36.617497 1236 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:60591 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:36.735961 1810 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:36.736258 1810 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:56:36.736517 1810 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:36.750489 1810 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:36.750641 1810 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:36.750824 1810 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:56:36.751186 1810 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:56:36.773355 1810 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50742 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:36.783643 1810 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:36.786510 1810 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 193678us May 06 07:56:36 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:36.791977 1810 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:56:36.797623 1810 fs_manager.cc:260] Metadata directory not provided I0506 07:56:36.797778 1810 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:36.798009 1810 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:56:36.798156 1810 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:36.808874 1810 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "da5642c3083249edbafecfee0027ce2a" format_stamp: "Formatted at 2018-05-06 14:56:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:36.809643 1810 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "da5642c3083249edbafecfee0027ce2a" format_stamp: "Formatted at 2018-05-06 14:56:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:36.821833 1810 fs_manager.cc:495] Time spent creating directory manager: real 0.012s user 0.009s sys 0.002s I0506 07:56:36.823065 1810 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:36.824398 1810 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:36.832876 1810 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0506 07:56:36.833083 1810 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/wal uuid: "da5642c3083249edbafecfee0027ce2a" format_stamp: "Formatted at 2018-05-06 14:56:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:36.833384 1810 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:37.044209 1810 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:37.049343 1810 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:37.051585 1810 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:37.177186 1810 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:40773 I0506 07:56:37.178272 1810 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:56:37.178390 1810 webserver.cc:184] Document root disabled I0506 07:56:37.180050 1810 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:34630/ I0506 07:56:37.180826 1810 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:56:37.185843 1810 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:56:37.191792 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 1810 May 06 07:56:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:56:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:37.229154 1938 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.194019 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:33736 0506 07:56:37.195804 (+ 1785us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.195828 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.197990 (+ 2162us) server_negotiation.cc:373] Connection header received 0506 07:56:37.207218 (+ 9228us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.207242 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.207322 (+ 80us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.207467 (+ 145us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:37.210681 (+ 3214us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.211085 (+ 404us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.212571 (+ 1486us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.213495 (+ 924us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.213696 (+ 201us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.217864 (+ 4168us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:37.217883 (+ 19us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:37.217896 (+ 13us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:37.217944 (+ 48us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:37.222317 (+ 4373us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.223584 (+ 1267us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.223624 (+ 40us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.223634 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.224107 (+ 473us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.224544 (+ 437us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.224561 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.224572 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.224912 (+ 340us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:37.225269 (+ 357us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.226055 (+ 786us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.226515 (+ 460us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":1573,"thread_start_us":1289,"threads_started":1} I0506 07:56:37.229344 1936 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.185094 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:50177 0506 07:56:37.186364 (+ 1270us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.186381 (+ 17us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.188827 (+ 2446us) server_negotiation.cc:373] Connection header received 0506 07:56:37.200867 (+ 12040us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.200897 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.201012 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.201253 (+ 241us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:37.204018 (+ 2765us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.204534 (+ 516us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.206028 (+ 1494us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.206927 (+ 899us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.207090 (+ 163us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.210703 (+ 3613us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:37.210753 (+ 50us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:37.210769 (+ 16us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:37.210820 (+ 51us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:37.216593 (+ 5773us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.218480 (+ 1887us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.218497 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.218508 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.222159 (+ 3651us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.223222 (+ 1063us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.223240 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.223250 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.223564 (+ 314us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:37.223944 (+ 380us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.225227 (+ 1283us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.225972 (+ 745us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2912,"server-negotiator.queue_time_us":1108,"thread_start_us":868,"threads_started":1} I0506 07:56:37.230777 1937 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.186043 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50742 0506 07:56:37.188644 (+ 2601us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.188725 (+ 81us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.200441 (+ 11716us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.202055 (+ 1614us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.202090 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.202985 (+ 895us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:37.203304 (+ 319us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.203726 (+ 422us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.203776 (+ 50us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.204765 (+ 989us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.204783 (+ 18us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.205813 (+ 1030us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.205847 (+ 34us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.207122 (+ 1275us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.207153 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.207531 (+ 378us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.207792 (+ 261us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:37.207805 (+ 13us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:37.210163 (+ 2358us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:37.217083 (+ 6920us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:37.217102 (+ 19us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:37.217117 (+ 15us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:37.218027 (+ 910us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:37.222565 (+ 4538us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:37.222582 (+ 17us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:37.222593 (+ 11us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:37.222789 (+ 196us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:37.224056 (+ 1267us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:37.224074 (+ 18us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:37.224383 (+ 309us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.224869 (+ 486us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.225236 (+ 367us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2306,"thread_start_us":1487,"threads_started":1} I0506 07:56:37.232981 1934 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50742 I0506 07:56:37.233275 1934 heartbeater.cc:426] Registering TS with master... I0506 07:56:37.233850 1934 heartbeater.cc:475] Master 127.0.196.126:50742 requested a full tablet report, sending... I0506 07:56:37.235414 1236 ts_manager.cc:81] Registered new tserver with Master: da5642c3083249edbafecfee0027ce2a (127.0.196.67:40773) I0506 07:56:37.236575 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0506 07:56:37.236647 1236 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:50177 May 06 07:56:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-user@KRBTEST.COM: May 06 07:56:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:37.288045 1936 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.268413 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:33737 0506 07:56:37.268788 (+ 375us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.268813 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.270085 (+ 1272us) server_negotiation.cc:373] Connection header received 0506 07:56:37.270264 (+ 179us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.270280 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.270366 (+ 86us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.270461 (+ 95us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:37.271875 (+ 1414us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.272258 (+ 383us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.273480 (+ 1222us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.274833 (+ 1353us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.275025 (+ 192us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.278665 (+ 3640us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:37.278686 (+ 21us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:37.278708 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:37.278763 (+ 55us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:37.282952 (+ 4189us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.284076 (+ 1124us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.284093 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.284104 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.284682 (+ 578us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:37.285073 (+ 391us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:37.285089 (+ 16us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:37.285100 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:37.285401 (+ 301us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:37.285682 (+ 281us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.286048 (+ 366us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.286752 (+ 704us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":158} I0506 07:56:37.294435 1236 catalog_manager.cc:1316] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.196.126:33737: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0506 07:56:37.303885 1236 catalog_manager.cc:1472] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0506 07:56:37.333688 1952 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.318329 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:46226 0506 07:56:37.320004 (+ 1675us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.320107 (+ 103us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.320345 (+ 238us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.321271 (+ 926us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.321302 (+ 31us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.321365 (+ 63us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:56:37.321846 (+ 481us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.322095 (+ 249us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.322140 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.323760 (+ 1620us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.323788 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.326659 (+ 2871us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.326702 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.330095 (+ 3393us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.330123 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.331469 (+ 1346us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.331633 (+ 164us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.331801 (+ 168us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.332232 (+ 431us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1417,"mutex_wait_us":750,"thread_start_us":1608,"threads_started":1} I0506 07:56:37.333986 1951 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.318497 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:42230 0506 07:56:37.319751 (+ 1254us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.319774 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.320709 (+ 935us) server_negotiation.cc:373] Connection header received 0506 07:56:37.320874 (+ 165us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.320895 (+ 21us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.321025 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.321174 (+ 149us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:56:37.322317 (+ 1143us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.323015 (+ 698us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.326892 (+ 3877us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.329849 (+ 2957us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.330049 (+ 200us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.330425 (+ 376us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.331907 (+ 1482us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.332256 (+ 349us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1083,"thread_start_us":834,"threads_started":1} I0506 07:56:37.337596 1485 tablet_service.cc:760] Processing CreateTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table=test-table [id=83368debe7cf4a1d94e819c88289390b]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:56:37.338546 1485 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14596c9b1dd04d569aa3c85cb0b6cc24. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:37.338580 1953 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.320801 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:58246 0506 07:56:37.323504 (+ 2703us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.323549 (+ 45us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.323733 (+ 184us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.326813 (+ 3080us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.326840 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.326899 (+ 59us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:56:37.327139 (+ 240us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.327499 (+ 360us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.327646 (+ 147us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.329107 (+ 1461us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.329136 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.332031 (+ 2895us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.332087 (+ 56us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.336194 (+ 4107us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.336224 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.336678 (+ 454us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.336823 (+ 145us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.336925 (+ 102us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.337230 (+ 305us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2503,"mutex_wait_us":104,"thread_start_us":2258,"threads_started":1} I0506 07:56:37.340617 1955 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.324436 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:40773 0506 07:56:37.328354 (+ 3918us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.328396 (+ 42us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.328522 (+ 126us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.329109 (+ 587us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.329134 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.329193 (+ 59us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:56:37.329467 (+ 274us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.329685 (+ 218us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.329730 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.331225 (+ 1495us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.331252 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.336107 (+ 4855us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.336153 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.339064 (+ 2911us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.339087 (+ 23us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.339364 (+ 277us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.339433 (+ 69us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.339507 (+ 74us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.339713 (+ 206us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3730,"mutex_wait_us":50,"thread_start_us":3016,"threads_started":1} I0506 07:56:37.340616 1954 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.321158 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:48426 0506 07:56:37.323991 (+ 2833us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.324023 (+ 32us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.324073 (+ 50us) server_negotiation.cc:373] Connection header received 0506 07:56:37.324232 (+ 159us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.326110 (+ 1878us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.326555 (+ 445us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.326744 (+ 189us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:56:37.328180 (+ 1436us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.328859 (+ 679us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.332346 (+ 3487us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.335428 (+ 3082us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.335643 (+ 215us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.336008 (+ 365us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.339046 (+ 3038us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.339375 (+ 329us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1692,"threads_started":1} I0506 07:56:37.341964 1956 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.324469 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:34866 0506 07:56:37.326587 (+ 2118us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.326617 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.328543 (+ 1926us) server_negotiation.cc:373] Connection header received 0506 07:56:37.328759 (+ 216us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.328777 (+ 18us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.328925 (+ 148us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.329060 (+ 135us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:56:37.330216 (+ 1156us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.330960 (+ 744us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.336433 (+ 5473us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.338484 (+ 2051us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.338629 (+ 145us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.338879 (+ 250us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.340695 (+ 1816us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.341169 (+ 474us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":1515,"threads_started":1} I0506 07:56:37.343299 1740 tablet_service.cc:760] Processing CreateTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table=test-table [id=83368debe7cf4a1d94e819c88289390b]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:56:37.344292 1740 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14596c9b1dd04d569aa3c85cb0b6cc24. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:37.344152 1871 tablet_service.cc:760] Processing CreateTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table=test-table [id=83368debe7cf4a1d94e819c88289390b]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:56:37.344884 1871 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14596c9b1dd04d569aa3c85cb0b6cc24. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:37.360015 1485 ts_tablet_manager.cc:1173] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Registered tablet (data state: TABLET_DATA_READY) I0506 07:56:37.360765 1871 ts_tablet_manager.cc:1173] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Registered tablet (data state: TABLET_DATA_READY) I0506 07:56:37.364020 1740 ts_tablet_manager.cc:1173] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Registered tablet (data state: TABLET_DATA_READY) I0506 07:56:37.365679 1958 ts_tablet_manager.cc:1028] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Bootstrapping tablet I0506 07:56:37.366401 1958 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Bootstrap starting. I0506 07:56:37.366457 1957 ts_tablet_manager.cc:1028] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Bootstrapping tablet I0506 07:56:37.367240 1957 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Bootstrap starting. I0506 07:56:37.367718 1959 ts_tablet_manager.cc:1028] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Bootstrapping tablet I0506 07:56:37.369256 1959 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Bootstrap starting. I0506 07:56:37.369814 1958 tablet_bootstrap.cc:587] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: No blocks or log segments found. Creating new log. I0506 07:56:37.370416 1957 tablet_bootstrap.cc:587] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: No blocks or log segments found. Creating new log. I0506 07:56:37.371546 1958 log.cc:520] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Log is configured to *not* fsync() on all Append() calls I0506 07:56:37.371639 1957 log.cc:520] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Log is configured to *not* fsync() on all Append() calls I0506 07:56:37.373461 1959 tablet_bootstrap.cc:587] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: No blocks or log segments found. Creating new log. I0506 07:56:37.375471 1957 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: No bootstrap required, opened a new log I0506 07:56:37.375722 1959 log.cc:520] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Log is configured to *not* fsync() on all Append() calls I0506 07:56:37.375964 1957 ts_tablet_manager.cc:1045] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s I0506 07:56:37.382621 1958 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: No bootstrap required, opened a new log I0506 07:56:37.383066 1958 ts_tablet_manager.cc:1045] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.008s I0506 07:56:37.386126 1959 tablet_bootstrap.cc:436] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: No bootstrap required, opened a new log I0506 07:56:37.386571 1959 ts_tablet_manager.cc:1045] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s I0506 07:56:37.403759 1957 raft_consensus.cc:303] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.404723 1957 raft_consensus.cc:329] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:37.404975 1957 raft_consensus.cc:604] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc96c3eb634481883f01cbea31443b2, State: Initialized, Role: FOLLOWER I0506 07:56:37.405858 1957 consensus_queue.cc:227] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.405797 1959 raft_consensus.cc:303] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.406806 1959 raft_consensus.cc:329] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:37.407049 1959 raft_consensus.cc:604] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243df4b1bb1e4bc98a379649d4c98796, State: Initialized, Role: FOLLOWER I0506 07:56:37.407835 1959 consensus_queue.cc:227] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.408181 1958 raft_consensus.cc:303] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.409112 1958 raft_consensus.cc:329] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:37.409353 1958 raft_consensus.cc:604] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da5642c3083249edbafecfee0027ce2a, State: Initialized, Role: FOLLOWER I0506 07:56:37.409950 1958 consensus_queue.cc:227] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.411090 1957 ts_tablet_manager.cc:1073] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Time spent starting tablet: real 0.035s user 0.020s sys 0.005s I0506 07:56:37.415019 1804 heartbeater.cc:467] Master 127.0.196.126:50742 was elected leader, sending a full tablet report... I0506 07:56:37.415947 1959 ts_tablet_manager.cc:1073] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Time spent starting tablet: real 0.029s user 0.018s sys 0.008s I0506 07:56:37.417330 1934 heartbeater.cc:467] Master 127.0.196.126:50742 was elected leader, sending a full tablet report... I0506 07:56:37.419025 1958 ts_tablet_manager.cc:1073] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Time spent starting tablet: real 0.036s user 0.021s sys 0.003s W0506 07:56:37.434576 1935 tablet.cc:1678] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:56:37.557497 1964 raft_consensus.cc:435] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0506 07:56:37.557734 1964 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Snoozing failure detection for 1.659s (starting election) I0506 07:56:37.557962 1964 raft_consensus.cc:457] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.558909 1964 leader_election.cc:230] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 243df4b1bb1e4bc98a379649d4c98796 I0506 07:56:37.559334 1964 leader_election.cc:230] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer da5642c3083249edbafecfee0027ce2a I0506 07:56:37.572777 1954 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.559858 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:41619 0506 07:56:37.560140 (+ 282us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.560164 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.561279 (+ 1115us) server_negotiation.cc:373] Connection header received 0506 07:56:37.561541 (+ 262us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.561567 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.561715 (+ 148us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.561826 (+ 111us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:56:37.562638 (+ 812us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.563062 (+ 424us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.565697 (+ 2635us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.567353 (+ 1656us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.567567 (+ 214us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.567851 (+ 284us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.571579 (+ 3728us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.571685 (+ 106us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":90} I0506 07:56:37.573835 1978 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.559682 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:58246 0506 07:56:37.561103 (+ 1421us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.561175 (+ 72us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.561353 (+ 178us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.561931 (+ 578us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.561957 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.562013 (+ 56us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:56:37.562229 (+ 216us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.562429 (+ 200us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.562459 (+ 30us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.563297 (+ 838us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.563324 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.565458 (+ 2134us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.565493 (+ 35us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.569007 (+ 3514us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.569037 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.570846 (+ 1809us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.571015 (+ 169us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.571380 (+ 365us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.571450 (+ 70us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1191,"mutex_wait_us":116,"thread_start_us":947,"threads_started":1} I0506 07:56:37.573922 1979 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.561167 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:40773 0506 07:56:37.562855 (+ 1688us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:37.562885 (+ 30us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:37.562982 (+ 97us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:37.563497 (+ 515us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:37.563513 (+ 16us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:37.563548 (+ 35us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:56:37.563761 (+ 213us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.563997 (+ 236us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.564025 (+ 28us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.564899 (+ 874us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.564914 (+ 15us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.566459 (+ 1545us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:37.566491 (+ 32us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.570471 (+ 3980us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.570499 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:37.571107 (+ 608us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.571300 (+ 193us) client_negotiation.cc:695] Sending connection context 0506 07:56:37.571631 (+ 331us) client_negotiation.cc:225] Negotiation successful 0506 07:56:37.571718 (+ 87us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":41,"thread_start_us":1157,"threads_started":1} I0506 07:56:37.574625 1956 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.561431 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:48234 0506 07:56:37.561788 (+ 357us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.561812 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.562960 (+ 1148us) server_negotiation.cc:373] Connection header received 0506 07:56:37.563203 (+ 243us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.563228 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.563357 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.563485 (+ 128us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:56:37.564184 (+ 699us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.564763 (+ 579us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.566681 (+ 1918us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.569011 (+ 2330us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.569196 (+ 185us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.569471 (+ 275us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.573508 (+ 4037us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.573581 (+ 73us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} I0506 07:56:37.576102 1760 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "14596c9b1dd04d569aa3c85cb0b6cc24" candidate_uuid: "9bc96c3eb634481883f01cbea31443b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243df4b1bb1e4bc98a379649d4c98796" is_pre_election: true I0506 07:56:37.576757 1760 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Snoozing failure detection for 1.655s (vote granted) I0506 07:56:37.576967 1891 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "14596c9b1dd04d569aa3c85cb0b6cc24" candidate_uuid: "9bc96c3eb634481883f01cbea31443b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5642c3083249edbafecfee0027ce2a" is_pre_election: true I0506 07:56:37.577040 1760 raft_consensus.cc:2219] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc96c3eb634481883f01cbea31443b2 in term 0. I0506 07:56:37.577456 1891 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Snoozing failure detection for 2.197s (vote granted) I0506 07:56:37.577699 1891 raft_consensus.cc:2219] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc96c3eb634481883f01cbea31443b2 in term 0. I0506 07:56:37.578560 1377 leader_election.cc:362] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 243df4b1bb1e4bc98a379649d4c98796 I0506 07:56:37.578824 1377 leader_election.cc:257] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0506 07:56:37.579351 1964 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Snoozing failure detection for 2.176s (election complete) I0506 07:56:37.579598 1964 raft_consensus.cc:2511] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0506 07:56:37.579864 1964 raft_consensus.cc:435] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0506 07:56:37.580077 1964 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Snoozing failure detection for 1.801s (starting election) I0506 07:56:37.580293 1964 raft_consensus.cc:2756] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:37.584980 1964 raft_consensus.cc:457] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.586170 1964 leader_election.cc:230] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 election: Requesting vote from peer 243df4b1bb1e4bc98a379649d4c98796 I0506 07:56:37.586561 1964 leader_election.cc:230] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 election: Requesting vote from peer da5642c3083249edbafecfee0027ce2a W0506 07:56:37.568804 1805 tablet.cc:1678] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:56:37.584133 1377 leader_election.cc:362] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 pre-election: Vote granted by peer da5642c3083249edbafecfee0027ce2a W0506 07:56:37.661783 1377 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.196.67:40773, user_credentials={real_user=jenkins}} blocked reactor thread for 77665us I0506 07:56:37.664002 1760 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "14596c9b1dd04d569aa3c85cb0b6cc24" candidate_uuid: "9bc96c3eb634481883f01cbea31443b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "243df4b1bb1e4bc98a379649d4c98796" I0506 07:56:37.664363 1760 raft_consensus.cc:2756] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:37.664307 1891 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "14596c9b1dd04d569aa3c85cb0b6cc24" candidate_uuid: "9bc96c3eb634481883f01cbea31443b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da5642c3083249edbafecfee0027ce2a" I0506 07:56:37.664597 1760 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Snoozing failure detection for 1.875s (vote granted) I0506 07:56:37.664783 1891 raft_consensus.cc:2756] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:37.665072 1891 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Snoozing failure detection for 1.989s (vote granted) I0506 07:56:37.667528 1760 raft_consensus.cc:2219] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc96c3eb634481883f01cbea31443b2 in term 1. I0506 07:56:37.668501 1377 leader_election.cc:362] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 election: Vote granted by peer 243df4b1bb1e4bc98a379649d4c98796 I0506 07:56:37.668697 1377 leader_election.cc:257] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:37.669077 1964 raft_consensus.cc:2704] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Snoozing failure detection for 1.588s (election complete) I0506 07:56:37.669296 1964 raft_consensus.cc:2511] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:37.669528 1891 raft_consensus.cc:2219] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc96c3eb634481883f01cbea31443b2 in term 1. I0506 07:56:37.670852 1377 leader_election.cc:362] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [CANDIDATE]: Term 1 election: Vote granted by peer da5642c3083249edbafecfee0027ce2a W0506 07:56:37.619383 1589 tablet.cc:1678] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:56:37.689620 1964 raft_consensus.cc:571] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc96c3eb634481883f01cbea31443b2, State: Running, Role: LEADER I0506 07:56:37.690205 1964 consensus_queue.cc:204] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } } I0506 07:56:37.696463 1236 catalog_manager.cc:3716] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 reported cstate change: term changed from 0 to 1, leader changed from to 9bc96c3eb634481883f01cbea31443b2 (127.0.196.65). New cstate: current_term: 1 leader_uuid: "9bc96c3eb634481883f01cbea31443b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bc96c3eb634481883f01cbea31443b2" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 46226 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 } health_report { overall_health: UNKNOWN } } } I0506 07:56:37.720182 1951 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:37.713790 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:42239 0506 07:56:37.714016 (+ 226us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:37.714033 (+ 17us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:37.714139 (+ 106us) server_negotiation.cc:373] Connection header received 0506 07:56:37.714411 (+ 272us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:37.714427 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:37.714527 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:37.714638 (+ 111us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:56:37.715090 (+ 452us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.715516 (+ 426us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.717051 (+ 1535us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:37.717948 (+ 897us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:37.718075 (+ 127us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:37.718769 (+ 694us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:56:37.719028 (+ 259us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:56:37.719117 (+ 89us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:37.719280 (+ 163us) server_negotiation.cc:264] Negotiation successful 0506 07:56:37.719409 (+ 129us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":80} I0506 07:56:37.729490 1760 raft_consensus.cc:1057] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 1 FOLLOWER]: Refusing update from remote peer 9bc96c3eb634481883f01cbea31443b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:56:37.729998 1891 raft_consensus.cc:1057] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 1 FOLLOWER]: Refusing update from remote peer 9bc96c3eb634481883f01cbea31443b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:56:37.730914 1964 consensus_queue.cc:963] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "243df4b1bb1e4bc98a379649d4c98796" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 58246 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:56:37.731947 1980 consensus_queue.cc:963] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da5642c3083249edbafecfee0027ce2a" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 40773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:56:37.776276 1235 catalog_manager.cc:1713] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.196.126:33737: table { table_name: "test-table" } I0506 07:56:37.782552 1235 catalog_manager.cc:3927] T 00000000000000000000000000000000 P 0bf0e56042e94995852891e9b599f500: Sending DeleteTablet for 3 replicas of tablet 14596c9b1dd04d569aa3c85cb0b6cc24 I0506 07:56:37.782833 1235 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 on 9bc96c3eb634481883f01cbea31443b2 (127.0.196.65:46226) (Table deleted at 2018-05-06 07:56:37 PDT) I0506 07:56:37.783215 1235 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 on 243df4b1bb1e4bc98a379649d4c98796 (127.0.196.66:58246) (Table deleted at 2018-05-06 07:56:37 PDT) I0506 07:56:37.783923 1485 tablet_service.cc:801] Processing DeleteTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:56:37 PDT) from {username='kudu', principal='kudu/127.0.196.126@KRBTEST.COM'} at 127.0.196.65:42230 I0506 07:56:37.783707 1235 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 on da5642c3083249edbafecfee0027ce2a (127.0.196.67:40773) (Table deleted at 2018-05-06 07:56:37 PDT) I0506 07:56:37.784371 1740 tablet_service.cc:801] Processing DeleteTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:56:37 PDT) from {username='kudu', principal='kudu/127.0.196.126@KRBTEST.COM'} at 127.0.196.66:48426 I0506 07:56:37.785497 1871 tablet_service.cc:801] Processing DeleteTablet for tablet 14596c9b1dd04d569aa3c85cb0b6cc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:56:37 PDT) from {username='kudu', principal='kudu/127.0.196.126@KRBTEST.COM'} at 127.0.196.67:34866 I0506 07:56:37.786942 1993 tablet_replica.cc:261] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: stopping tablet replica I0506 07:56:37.787374 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op CompactRowSetsOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.787627 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op MinorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.787883 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op MajorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.788228 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op UndoDeltaBlockGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.788496 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op FlushMRSOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.788734 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op FlushDeltaMemStoresOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.788959 1993 maintenance_manager.cc:237] P 9bc96c3eb634481883f01cbea31443b2: Unregistered op LogGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.789508 1993 raft_consensus.cc:2016] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 1 LEADER]: Raft consensus shutting down. I0506 07:56:37.795158 1995 tablet_replica.cc:261] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: stopping tablet replica I0506 07:56:37.795538 1994 tablet_replica.cc:261] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: stopping tablet replica I0506 07:56:37.795586 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op CompactRowSetsOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.795871 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op CompactRowSetsOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.795917 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op MinorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796079 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op MinorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796170 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op MajorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796268 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op MajorDeltaCompactionOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796445 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op UndoDeltaBlockGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796442 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op UndoDeltaBlockGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796685 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op FlushMRSOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796748 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op FlushMRSOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.796866 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op FlushDeltaMemStoresOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.797034 1994 maintenance_manager.cc:237] P 243df4b1bb1e4bc98a379649d4c98796: Unregistered op LogGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.797021 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op FlushDeltaMemStoresOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.797236 1995 maintenance_manager.cc:237] P da5642c3083249edbafecfee0027ce2a: Unregistered op LogGCOp(14596c9b1dd04d569aa3c85cb0b6cc24) I0506 07:56:37.797236 1994 raft_consensus.cc:2016] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 1 FOLLOWER]: Raft consensus shutting down. I0506 07:56:37.797488 1995 raft_consensus.cc:2016] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 1 FOLLOWER]: Raft consensus shutting down. I0506 07:56:37.797544 1994 raft_consensus.cc:2043] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796 [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:56:37.797878 1995 raft_consensus.cc:2043] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:56:37.801079 1994 ts_tablet_manager.cc:1369] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:56:37.801314 1995 ts_tablet_manager.cc:1369] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:56:37.810344 1994 ts_tablet_manager.cc:1382] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:56:37.810655 1994 log.cc:981] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-1/wal/wals/14596c9b1dd04d569aa3c85cb0b6cc24 I0506 07:56:37.810770 1993 consensus_peers.cc:456] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 -> Peer da5642c3083249edbafecfee0027ce2a (127.0.196.67:40773): Closing peer: da5642c3083249edbafecfee0027ce2a I0506 07:56:37.811453 1994 ts_tablet_manager.cc:1402] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 243df4b1bb1e4bc98a379649d4c98796: Deleting consensus metadata I0506 07:56:37.814203 1170 catalog_manager.cc:3021] TS 243df4b1bb1e4bc98a379649d4c98796 (127.0.196.66:58246): tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table test-table [id=83368debe7cf4a1d94e819c88289390b]) successfully deleted I0506 07:56:37.813174 1993 consensus_peers.cc:456] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 -> Peer 243df4b1bb1e4bc98a379649d4c98796 (127.0.196.66:58246): Closing peer: 243df4b1bb1e4bc98a379649d4c98796 I0506 07:56:37.815712 1993 raft_consensus.cc:2043] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2 [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:56:37.815968 1995 ts_tablet_manager.cc:1382] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:56:37.816217 1995 log.cc:981] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-2/wal/wals/14596c9b1dd04d569aa3c85cb0b6cc24 I0506 07:56:37.816821 1995 ts_tablet_manager.cc:1402] T 14596c9b1dd04d569aa3c85cb0b6cc24 P da5642c3083249edbafecfee0027ce2a: Deleting consensus metadata I0506 07:56:37.818405 1993 ts_tablet_manager.cc:1369] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:56:37.819154 1170 catalog_manager.cc:3021] TS da5642c3083249edbafecfee0027ce2a (127.0.196.67:40773): tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table test-table [id=83368debe7cf4a1d94e819c88289390b]) successfully deleted I0506 07:56:37.828508 1993 ts_tablet_manager.cc:1382] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:56:37.828738 1993 log.cc:981] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1525618592290500-785/minicluster-data/ts-0/wal/wals/14596c9b1dd04d569aa3c85cb0b6cc24 I0506 07:56:37.829150 1993 ts_tablet_manager.cc:1402] T 14596c9b1dd04d569aa3c85cb0b6cc24 P 9bc96c3eb634481883f01cbea31443b2: Deleting consensus metadata I0506 07:56:37.830687 1170 catalog_manager.cc:3021] TS 9bc96c3eb634481883f01cbea31443b2 (127.0.196.65:46226): tablet 14596c9b1dd04d569aa3c85cb0b6cc24 (table test-table [id=83368debe7cf4a1d94e819c88289390b]) successfully deleted May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618597, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.196.65@KRBTEST.COM I0506 07:56:38.046366 1951 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:38.022437 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:42242 0506 07:56:38.022795 (+ 358us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:38.022816 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:38.024428 (+ 1612us) server_negotiation.cc:373] Connection header received 0506 07:56:38.024552 (+ 124us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:38.024578 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:38.024737 (+ 159us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:38.024893 (+ 156us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:38.026738 (+ 1845us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:38.027134 (+ 396us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:38.028412 (+ 1278us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:38.029318 (+ 906us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:38.029496 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:38.032386 (+ 2890us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:38.032409 (+ 23us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:38.032426 (+ 17us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:38.032479 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:38.037788 (+ 5309us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:38.039115 (+ 1327us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:38.039137 (+ 22us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:38.039148 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:38.039799 (+ 651us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:38.040388 (+ 589us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:38.040415 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:38.040435 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:38.040990 (+ 555us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:38.043404 (+ 2414us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:38.043700 (+ 296us) server_negotiation.cc:264] Negotiation successful 0506 07:56:38.044193 (+ 493us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":126} W0506 07:56:38.047721 1578 server_base.cc:567] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.196.65:42242 I0506 07:56:38.102018 2002 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:38.067230 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:33751 0506 07:56:38.068761 (+ 1531us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:38.068792 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:38.068833 (+ 41us) server_negotiation.cc:373] Connection header received 0506 07:56:38.068954 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:38.068977 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:38.069108 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:38.071265 (+ 2157us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:38.071504 (+ 239us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:38.072269 (+ 765us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:38.075072 (+ 2803us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:38.077905 (+ 2833us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:38.080906 (+ 3001us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:38.081205 (+ 299us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:38.081231 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:38.081253 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:38.081324 (+ 71us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:38.088855 (+ 7531us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:38.092955 (+ 4100us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:38.092982 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:38.093003 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:38.093687 (+ 684us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:38.095329 (+ 1642us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:38.095356 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:38.095375 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:38.095941 (+ 566us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:38.098586 (+ 2645us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:38.098821 (+ 235us) server_negotiation.cc:264] Negotiation successful 0506 07:56:38.099702 (+ 881us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":1007,"threads_started":1} W0506 07:56:38.103021 1235 server_base.cc:567] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.196.126:33751 I0506 07:56:38.104547 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 1283 I0506 07:56:38.148062 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 1600 W0506 07:56:38.148082 1217 connection.cc:512] server connection from 127.0.196.65:39934 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:39934): Connection reset by peer (error 104) W0506 07:56:38.152294 1821 connection.cc:512] server connection from 127.0.196.65:48234 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:48234): Connection reset by peer (error 104) W0506 07:56:38.173722 1217 connection.cc:512] server connection from 127.0.196.66:60591 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:60591): Connection reset by peer (error 104) I0506 07:56:38.174419 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 1810 I0506 07:56:38.202998 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 1027 [ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5813 ms) [ RUN ] SecurityITest.TestNoKerberosCredentials Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): setting up network... May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): skipping unrecognized local address family 17 May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): skipping unrecognized local address family 17 May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): set up 1 sockets May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): commencing operation krb5kdc: starting... W0506 07:56:38.367189 785 mini_kdc.cc:117] Time spent starting KDC: real 0.110s user 0.002s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:38.491351 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:33803 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:38.658428 2065 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:38.658864 2065 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:38.659217 2065 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:38.670529 2065 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:38.670676 2065 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:38.670780 2065 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:38.670877 2065 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:38.671016 2065 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:38.671138 2065 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:38.692404 2065 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:33803 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:38.702073 2065 master_main.cc:73] Initializing master server... I0506 07:56:38.704524 2065 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 194678us May 06 07:56:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:38.709810 2065 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:38.714967 2065 fs_manager.cc:260] Metadata directory not provided I0506 07:56:38.715124 2065 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:38.715355 2065 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:38.715483 2065 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:38.731025 2065 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "2a6b08caee48425e9a97c78f7f8abe47" format_stamp: "Formatted at 2018-05-06 14:56:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:38.732173 2065 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "2a6b08caee48425e9a97c78f7f8abe47" format_stamp: "Formatted at 2018-05-06 14:56:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:38.750324 2065 fs_manager.cc:495] Time spent creating directory manager: real 0.018s user 0.013s sys 0.003s I0506 07:56:38.752192 2065 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:38.754209 2065 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:38.768128 2065 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0506 07:56:38.768306 2065 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal uuid: "2a6b08caee48425e9a97c78f7f8abe47" format_stamp: "Formatted at 2018-05-06 14:56:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:38.768649 2065 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:39.031682 2065 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:39.039610 2065 master_main.cc:76] Starting Master server... I0506 07:56:39.257712 2065 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:33803 I0506 07:56:39.258710 2065 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:56:39.258821 2065 webserver.cc:184] Document root disabled I0506 07:56:39.262091 2065 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:50018/ I0506 07:56:39.263479 2065 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:39.272485 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 2065 I0506 07:56:39.274380 2340 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:56:39.313412 2340 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Bootstrap starting. I0506 07:56:39.316491 2340 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: No blocks or log segments found. Creating new log. I0506 07:56:39.318724 2340 log.cc:520] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Log is configured to *not* fsync() on all Append() calls Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:56:39.321139 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:39.326601 2340 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: No bootstrap required, opened a new log I0506 07:56:39.347949 2340 raft_consensus.cc:303] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } I0506 07:56:39.348489 2340 raft_consensus.cc:329] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:39.348783 2340 raft_consensus.cc:604] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6b08caee48425e9a97c78f7f8abe47, State: Initialized, Role: FOLLOWER I0506 07:56:39.349453 2340 consensus_queue.cc:227] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } I0506 07:56:39.350081 2340 raft_consensus.cc:343] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:39.350379 2340 raft_consensus.cc:435] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:39.350661 2340 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Snoozing failure detection for 1.732s (starting election) I0506 07:56:39.350939 2340 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:39.355029 2340 raft_consensus.cc:457] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } I0506 07:56:39.355715 2340 leader_election.cc:257] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:39.362679 2359 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Snoozing failure detection for 1.789s (election complete) I0506 07:56:39.364856 2359 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:39.374897 2359 raft_consensus.cc:571] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [term 1 LEADER]: Becoming Leader. State: Replica: 2a6b08caee48425e9a97c78f7f8abe47, State: Running, Role: LEADER I0506 07:56:39.375941 2359 consensus_queue.cc:204] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } I0506 07:56:39.398118 2361 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6b08caee48425e9a97c78f7f8abe47. Latest consensus state: current_term: 1 leader_uuid: "2a6b08caee48425e9a97c78f7f8abe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } } I0506 07:56:39.398572 2361 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: This master's current role is: LEADER I0506 07:56:39.400056 2340 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "2a6b08caee48425e9a97c78f7f8abe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } } I0506 07:56:39.524869 2340 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: This master's current role is: LEADER I0506 07:56:39.526295 2340 sys_catalog.cc:422] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:39.400310 2360 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a6b08caee48425e9a97c78f7f8abe47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a6b08caee48425e9a97c78f7f8abe47" member_type: VOTER } } I0506 07:56:39.536083 2360 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47 [sys.catalog]: This master's current role is: LEADER I0506 07:56:39.539636 2065 master_main.cc:79] Master server successfully started. I0506 07:56:39.538794 2363 catalog_manager.cc:990] Loading table and tablet metadata into memory... W0506 07:56:39.547857 2239 tablet.cc:1678] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:56:39.549736 2363 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:39.587649 2355 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:39.588150 2355 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:56:39.588562 2355 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:39.603451 2355 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:39.603672 2355 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:39.603945 2355 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:56:39.604440 2355 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:56:39.635113 2355 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:39.652869 2355 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:39.655974 2355 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 195178us May 06 07:56:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:39.662261 2355 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:56:39.676041 2355 fs_manager.cc:260] Metadata directory not provided I0506 07:56:39.676265 2355 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:39.676612 2355 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:56:39.676831 2355 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:39.692322 2355 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "ce79bea1e0f1481b96405a217fe9b48c" format_stamp: "Formatted at 2018-05-06 14:56:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:39.693367 2355 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "ce79bea1e0f1481b96405a217fe9b48c" format_stamp: "Formatted at 2018-05-06 14:56:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:39.718152 2355 fs_manager.cc:495] Time spent creating directory manager: real 0.024s user 0.010s sys 0.010s I0506 07:56:39.720520 2355 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:39.723791 2355 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:39.749153 2355 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s I0506 07:56:39.749370 2355 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal uuid: "ce79bea1e0f1481b96405a217fe9b48c" format_stamp: "Formatted at 2018-05-06 14:56:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:39.749814 2355 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:39.777716 2363 catalog_manager.cc:888] Generated new certificate authority record I0506 07:56:39.779276 2363 catalog_manager.cc:1010] Loading token signing keys... I0506 07:56:39.791589 2363 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 2a6b08caee48425e9a97c78f7f8abe47: Generated new TSK 0 I0506 07:56:39.998775 2355 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:40.007746 2355 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:40.015877 2355 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:40.672250 2355 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:51693 I0506 07:56:40.673243 2355 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:56:40.673354 2355 webserver.cc:184] Document root disabled I0506 07:56:40.675942 2355 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:50643/ I0506 07:56:40.676964 2355 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:56:40.683878 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 2355 I0506 07:56:40.696887 2355 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. May 06 07:56:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:56:40.756397 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:40.756887 2769 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:40.690804 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:52119 0506 07:56:40.692454 (+ 1650us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:40.692496 (+ 42us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:40.702928 (+ 10432us) server_negotiation.cc:373] Connection header received 0506 07:56:40.721128 (+ 18200us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:40.721169 (+ 41us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:40.721403 (+ 234us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:40.721731 (+ 328us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:40.724071 (+ 2340us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:40.724899 (+ 828us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:40.726968 (+ 2069us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:40.729188 (+ 2220us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:40.729399 (+ 211us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:40.735903 (+ 6504us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:40.735977 (+ 74us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:40.736008 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:40.736086 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:40.743941 (+ 7855us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:40.746795 (+ 2854us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:40.746828 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:40.746850 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:40.747540 (+ 690us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:40.749119 (+ 1579us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:40.749144 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:40.749164 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:40.749776 (+ 612us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:40.750225 (+ 449us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:40.752229 (+ 2004us) server_negotiation.cc:264] Negotiation successful 0506 07:56:40.755008 (+ 2779us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1320,"thread_start_us":924,"threads_started":1} I0506 07:56:40.757805 2770 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:40.690826 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:33803 0506 07:56:40.701914 (+ 11088us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:40.702797 (+ 883us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:40.720502 (+ 17705us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:40.722292 (+ 1790us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:40.722325 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:40.723135 (+ 810us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:40.723419 (+ 284us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:40.723796 (+ 377us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:40.723846 (+ 50us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:40.725177 (+ 1331us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:40.725204 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:40.726733 (+ 1529us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:40.726777 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:40.729958 (+ 3181us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:40.729988 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:40.730441 (+ 453us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:40.730748 (+ 307us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:40.730768 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:40.735037 (+ 4269us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:40.744626 (+ 9589us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:40.744801 (+ 175us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:40.744831 (+ 30us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:40.746077 (+ 1246us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:40.748212 (+ 2135us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:40.748242 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:40.748262 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:40.748470 (+ 208us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:40.750607 (+ 2137us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:40.750634 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:40.751074 (+ 440us) client_negotiation.cc:695] Sending connection context 0506 07:56:40.751600 (+ 526us) client_negotiation.cc:225] Negotiation successful 0506 07:56:40.752950 (+ 1350us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10706,"thread_start_us":11208,"threads_started":1} I0506 07:56:40.761261 2763 heartbeater.cc:346] Connected to a master server at 127.0.196.126:33803 I0506 07:56:40.761551 2763 heartbeater.cc:426] Registering TS with master... I0506 07:56:40.762145 2763 heartbeater.cc:475] Master 127.0.196.126:33803 requested a full tablet report, sending... I0506 07:56:40.764500 2259 ts_manager.cc:81] Registered new tserver with Master: ce79bea1e0f1481b96405a217fe9b48c (127.0.196.65:51693) I0506 07:56:40.767347 2259 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:52119 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:40.945057 2780 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:40.945571 2780 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:56:40.946020 2780 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:40.966231 2780 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:40.966456 2780 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:40.966774 2780 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:56:40.967296 2780 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:56:41.008796 2780 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:41.024755 2780 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:41.031585 2780 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 196178us May 06 07:56:41 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:41.035868 2780 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:56:41.043915 2780 fs_manager.cc:260] Metadata directory not provided I0506 07:56:41.044139 2780 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:41.044502 2780 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:56:41.044843 2780 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:41.059666 2780 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "d2aff5a98ec54f18a49fd1b795c2327f" format_stamp: "Formatted at 2018-05-06 14:56:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:41.060737 2780 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "d2aff5a98ec54f18a49fd1b795c2327f" format_stamp: "Formatted at 2018-05-06 14:56:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:41.081082 2780 fs_manager.cc:495] Time spent creating directory manager: real 0.020s user 0.011s sys 0.006s I0506 07:56:41.083096 2780 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:41.083788 2780 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:41.107830 2780 fs_manager.cc:417] Time spent opening block manager: real 0.012s user 0.000s sys 0.005s I0506 07:56:41.108037 2780 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal uuid: "d2aff5a98ec54f18a49fd1b795c2327f" format_stamp: "Formatted at 2018-05-06 14:56:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:41.108408 2780 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:41.410893 2780 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:41.419343 2780 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:41.432175 2780 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:41.771387 2763 heartbeater.cc:467] Master 127.0.196.126:33803 was elected leader, sending a full tablet report... I0506 07:56:41.971384 2780 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:53855 I0506 07:56:41.972432 2780 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:56:41.972561 2780 webserver.cc:184] Document root disabled I0506 07:56:41.980000 2780 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:36730/ I0506 07:56:41.981055 2780 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:56:41.987511 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 2780 WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. I0506 07:56:42.013909 2780 tablet_server_main.cc:87] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. I0506 07:56:42.034926 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:56:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:42.061303 3110 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:41.995535 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:52686 0506 07:56:42.004283 (+ 8748us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:42.004309 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:42.012186 (+ 7877us) server_negotiation.cc:373] Connection header received 0506 07:56:42.028998 (+ 16812us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:42.029028 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:42.029158 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:42.029345 (+ 187us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:42.031817 (+ 2472us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.032483 (+ 666us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.034762 (+ 2279us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.036228 (+ 1466us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.036392 (+ 164us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:42.041800 (+ 5408us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:42.041868 (+ 68us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:42.041893 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:42.041970 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:42.049336 (+ 7366us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.052355 (+ 3019us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.052382 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.052401 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.053074 (+ 673us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.054721 (+ 1647us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.054750 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.054769 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.055297 (+ 528us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:42.055669 (+ 372us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:42.059286 (+ 3617us) server_negotiation.cc:264] Negotiation successful 0506 07:56:42.059608 (+ 322us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8551,"thread_start_us":8309,"threads_started":1} I0506 07:56:42.063441 3111 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:41.995846 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:33803 0506 07:56:42.011585 (+ 15739us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:42.012048 (+ 463us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:42.028736 (+ 16688us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:42.029877 (+ 1141us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:42.029909 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:42.030793 (+ 884us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:42.031084 (+ 291us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.031522 (+ 438us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:42.031574 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.032787 (+ 1213us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.032813 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.034491 (+ 1678us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:42.034532 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.036850 (+ 2318us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.036878 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.037304 (+ 426us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:42.037598 (+ 294us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:42.037618 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:42.040976 (+ 3358us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:42.050161 (+ 9185us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:42.050192 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:42.050218 (+ 26us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:42.051607 (+ 1389us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:42.053747 (+ 2140us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:42.053773 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:42.053794 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:42.054042 (+ 248us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:42.056089 (+ 2047us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:42.056122 (+ 33us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:42.056599 (+ 477us) client_negotiation.cc:695] Sending connection context 0506 07:56:42.057389 (+ 790us) client_negotiation.cc:225] Negotiation successful 0506 07:56:42.057819 (+ 430us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15376,"thread_start_us":14568,"threads_started":1} I0506 07:56:42.067176 3107 heartbeater.cc:346] Connected to a master server at 127.0.196.126:33803 I0506 07:56:42.067510 3107 heartbeater.cc:426] Registering TS with master... I0506 07:56:42.068219 3107 heartbeater.cc:475] Master 127.0.196.126:33803 requested a full tablet report, sending... I0506 07:56:42.070366 2259 ts_manager.cc:81] Registered new tserver with Master: d2aff5a98ec54f18a49fd1b795c2327f (127.0.196.66:53855) I0506 07:56:42.072255 2259 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:52686 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:42.269948 3120 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:42.270450 3120 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:56:42.270912 3120 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:42.295439 3120 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:42.295660 3120 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:42.295984 3120 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:56:42.296555 3120 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:56:42.343940 3120 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:33803 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:42.358070 3120 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:42.359988 3120 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 196678us May 06 07:56:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:42.363183 3120 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:56:42.367574 3120 fs_manager.cc:260] Metadata directory not provided I0506 07:56:42.367743 3120 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:42.367974 3120 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:56:42.368118 3120 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:42.378167 3120 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "dd633df4f270496d88c4bb81526a414b" format_stamp: "Formatted at 2018-05-06 14:56:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:42.378973 3120 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "dd633df4f270496d88c4bb81526a414b" format_stamp: "Formatted at 2018-05-06 14:56:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:42.396863 3120 fs_manager.cc:495] Time spent creating directory manager: real 0.017s user 0.013s sys 0.002s I0506 07:56:42.405887 3120 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:42.406512 3120 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:42.420907 3120 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.000s sys 0.007s I0506 07:56:42.421031 3120 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal uuid: "dd633df4f270496d88c4bb81526a414b" format_stamp: "Formatted at 2018-05-06 14:56:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:42.421267 3120 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:42.645848 3120 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:42.651144 3120 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:42.653343 3120 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:42.888777 3120 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:60504 I0506 07:56:42.889984 3120 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:56:42.890108 3120 webserver.cc:184] Document root disabled I0506 07:56:42.901638 3120 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:48307/ I0506 07:56:42.902766 3120 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:56:42.907320 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 3120 I0506 07:56:42.927817 3120 tablet_server_main.cc:87] Tablet server successfully started. May 06 07:56:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:42.957430 3404 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:42.909137 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:59862 0506 07:56:42.914939 (+ 5802us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:42.914964 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:42.919645 (+ 4681us) server_negotiation.cc:373] Connection header received 0506 07:56:42.919975 (+ 330us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:42.920006 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:42.920128 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:42.920325 (+ 197us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:42.923183 (+ 2858us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.924440 (+ 1257us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.926440 (+ 2000us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.927933 (+ 1493us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.928101 (+ 168us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:42.941311 (+ 13210us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:42.941340 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:42.941364 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:42.941448 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:42.948592 (+ 7144us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.950395 (+ 1803us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.950425 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.950445 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.951549 (+ 1104us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.952201 (+ 652us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.952230 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.952250 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.952783 (+ 533us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:42.953154 (+ 371us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:42.955093 (+ 1939us) server_negotiation.cc:264] Negotiation successful 0506 07:56:42.955555 (+ 462us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5602,"thread_start_us":5377,"threads_started":1} May 06 07:56:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[2050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:42.985956 3407 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:42.917778 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:33803 0506 07:56:42.929411 (+ 11633us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:42.929489 (+ 78us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:42.945986 (+ 16497us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:42.948931 (+ 2945us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:42.948968 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:42.949829 (+ 861us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:42.950131 (+ 302us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.950543 (+ 412us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:42.950601 (+ 58us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.956782 (+ 6181us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.956813 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.958485 (+ 1672us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:42.958537 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.960535 (+ 1998us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.960565 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:42.961272 (+ 707us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:42.961603 (+ 331us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:42.961650 (+ 47us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:42.965101 (+ 3451us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:42.973641 (+ 8540us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:42.973672 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:42.973722 (+ 50us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:42.975199 (+ 1477us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:42.978307 (+ 3108us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:42.978334 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:42.978354 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:42.978618 (+ 264us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:42.981606 (+ 2988us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:42.981636 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:42.982309 (+ 673us) client_negotiation.cc:695] Sending connection context 0506 07:56:42.982994 (+ 685us) client_negotiation.cc:225] Negotiation successful 0506 07:56:42.983378 (+ 384us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":11138,"thread_start_us":13591,"threads_started":1} I0506 07:56:42.988811 3408 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:42.919826 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:39292 0506 07:56:42.923777 (+ 3951us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:42.923804 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:42.929638 (+ 5834us) server_negotiation.cc:373] Connection header received 0506 07:56:42.946268 (+ 16630us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:42.946293 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:42.946423 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:42.946592 (+ 169us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:42.950827 (+ 4235us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.951379 (+ 552us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.958819 (+ 7440us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:42.960278 (+ 1459us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:42.960451 (+ 173us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:42.965951 (+ 5500us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:42.966012 (+ 61us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:42.966034 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:42.966100 (+ 66us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:42.973117 (+ 7017us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.975721 (+ 2604us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.975746 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.975764 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.976981 (+ 1217us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:42.979689 (+ 2708us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:42.980140 (+ 451us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:42.980161 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:42.981079 (+ 918us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:42.981450 (+ 371us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:42.983675 (+ 2225us) server_negotiation.cc:264] Negotiation successful 0506 07:56:42.984007 (+ 332us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3680,"thread_start_us":3355,"threads_started":1} I0506 07:56:42.993269 3402 heartbeater.cc:346] Connected to a master server at 127.0.196.126:33803 I0506 07:56:42.993515 3402 heartbeater.cc:426] Registering TS with master... I0506 07:56:42.994014 3402 heartbeater.cc:475] Master 127.0.196.126:33803 requested a full tablet report, sending... I0506 07:56:42.996321 2259 ts_manager.cc:81] Registered new tserver with Master: dd633df4f270496d88c4bb81526a414b (127.0.196.67:60504) I0506 07:56:42.998280 2259 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:39292 I0506 07:56:43.004886 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters W0506 07:56:43.040623 3408 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:56:43.035213 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:59863 0506 07:56:43.035575 (+ 362us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:43.035597 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:43.038067 (+ 2470us) server_negotiation.cc:373] Connection header received 0506 07:56:43.038298 (+ 231us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:43.038314 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:43.038398 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:43.038504 (+ 106us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:43.040136 (+ 1632us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.126:59863: BlockingRecv error: recv got EOF from 127.0.196.126:59863 (error 108) Metrics: {"server-negotiator.queue_time_us":148} I0506 07:56:43.052078 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 2355 I0506 07:56:43.076231 3107 heartbeater.cc:467] Master 127.0.196.126:33803 was elected leader, sending a full tablet report... I0506 07:56:43.088938 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 2780 W0506 07:56:43.089166 2164 connection.cc:512] server connection from 127.0.196.65:52119 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:52119): Connection reset by peer (error 104) W0506 07:56:43.126062 2165 connection.cc:512] server connection from 127.0.196.66:52686 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:52686): Connection reset by peer (error 104) I0506 07:56:43.126866 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 3120 W0506 07:56:43.160501 2163 connection.cc:512] server connection from 127.0.196.67:39292 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:39292): Connection reset by peer (error 104) I0506 07:56:43.161406 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 2065 [ OK ] SecurityITest.TestNoKerberosCredentials (4970 ms) [ RUN ] SecurityITest.SaslPlainFallback Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): setting up network... May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): skipping unrecognized local address family 17 May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): skipping unrecognized local address family 17 May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): set up 1 sockets May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): commencing operation krb5kdc: starting... W0506 07:56:43.392257 785 mini_kdc.cc:117] Time spent starting KDC: real 0.161s user 0.005s sys 0.010s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:43 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:43.603497 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:45724 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc-authentication=optional --user-acl=* --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:43.883919 3515 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:43.884434 3515 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:43.884862 3515 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:43.911749 3515 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:43.912080 3515 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:43.912269 3515 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:43.912447 3515 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:43.912673 3515 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:43.912890 3515 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:43.981087 3515 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:45724 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:44.003906 3515 master_main.cc:73] Initializing master server... I0506 07:56:44.006831 3515 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 197678us May 06 07:56:44 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:44.018795 3515 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:44.039021 3515 fs_manager.cc:260] Metadata directory not provided I0506 07:56:44.039257 3515 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:44.039633 3515 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:44.040874 3515 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:44.068119 3515 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "6b6cb4d636034256bb11f126107bff9f" format_stamp: "Formatted at 2018-05-06 14:56:44 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:44.069941 3515 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "6b6cb4d636034256bb11f126107bff9f" format_stamp: "Formatted at 2018-05-06 14:56:44 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:44.118235 3515 fs_manager.cc:495] Time spent creating directory manager: real 0.048s user 0.011s sys 0.006s I0506 07:56:44.122349 3515 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:44.125538 3515 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:44.152712 3515 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.000s sys 0.006s I0506 07:56:44.152926 3515 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/wal uuid: "6b6cb4d636034256bb11f126107bff9f" format_stamp: "Formatted at 2018-05-06 14:56:44 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:44.153321 3515 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:44.652289 3515 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:44.666541 3515 master_main.cc:76] Starting Master server... I0506 07:56:44.986148 3515 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:45724 I0506 07:56:44.987571 3515 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:56:44.987776 3515 webserver.cc:184] Document root disabled I0506 07:56:44.994243 3515 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:36485/ I0506 07:56:44.995369 3515 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:44.999469 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 3515 I0506 07:56:45.010897 3739 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:45.047436 3739 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: Bootstrap starting. I0506 07:56:45.050783 3739 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: No blocks or log segments found. Creating new log. May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:45.063238 3739 log.cc:520] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: Log is configured to *not* fsync() on all Append() calls I0506 07:56:45.070951 3739 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: No bootstrap required, opened a new log I0506 07:56:45.080988 3744 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:45.005322 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34908 0506 07:56:45.016359 (+ 11037us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:45.016397 (+ 38us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:45.016494 (+ 97us) server_negotiation.cc:373] Connection header received 0506 07:56:45.041593 (+ 25099us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:45.041652 (+ 59us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:45.042092 (+ 440us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:45.042411 (+ 319us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:45.045236 (+ 2825us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:45.046034 (+ 798us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:45.047934 (+ 1900us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:45.050210 (+ 2276us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:45.054873 (+ 4663us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:45.055239 (+ 366us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:45.055274 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:45.055303 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:45.055389 (+ 86us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:45.071561 (+ 16172us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:45.073481 (+ 1920us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:45.073511 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:45.073533 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:45.074313 (+ 780us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:45.074965 (+ 652us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:45.074992 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:45.075012 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:45.075756 (+ 744us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:45.078089 (+ 2333us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:45.078375 (+ 286us) server_negotiation.cc:264] Negotiation successful 0506 07:56:45.078926 (+ 551us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10665,"thread_start_us":2114,"threads_started":1} I0506 07:56:45.083614 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:56:45.184801 3744 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:45.159644 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34914 0506 07:56:45.160005 (+ 361us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:45.160029 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:45.172747 (+ 12718us) server_negotiation.cc:373] Connection header received 0506 07:56:45.172865 (+ 118us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:45.172890 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:45.173021 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:45.173143 (+ 122us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:45.175617 (+ 2474us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:45.176135 (+ 518us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:45.178365 (+ 2230us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:45.180182 (+ 1817us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:45.180333 (+ 151us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:45.181584 (+ 1251us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:45.181608 (+ 24us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:45.181627 (+ 19us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:56:45.182224 (+ 597us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:45.182300 (+ 76us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:56:45.182362 (+ 62us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:45.182459 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:45.183342 (+ 883us) server_negotiation.cc:264] Negotiation successful 0506 07:56:45.183526 (+ 184us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":151} I0506 07:56:45.101438 3739 raft_consensus.cc:303] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } I0506 07:56:45.247067 3739 raft_consensus.cc:329] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:45.247295 3739 raft_consensus.cc:604] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6cb4d636034256bb11f126107bff9f, State: Initialized, Role: FOLLOWER I0506 07:56:45.248049 3739 consensus_queue.cc:227] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } I0506 07:56:45.248574 3739 raft_consensus.cc:343] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:45.248862 3739 raft_consensus.cc:435] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:45.249112 3739 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: Snoozing failure detection for 2.208s (starting election) I0506 07:56:45.249353 3739 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:45.254071 3739 raft_consensus.cc:457] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } I0506 07:56:45.254715 3739 leader_election.cc:257] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:45.269852 3834 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: Snoozing failure detection for 1.517s (election complete) I0506 07:56:45.270139 3834 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:45.280611 3834 raft_consensus.cc:571] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [term 1 LEADER]: Becoming Leader. State: Replica: 6b6cb4d636034256bb11f126107bff9f, State: Running, Role: LEADER I0506 07:56:45.281828 3834 consensus_queue.cc:204] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } I0506 07:56:45.297660 3840 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b6cb4d636034256bb11f126107bff9f. Latest consensus state: current_term: 1 leader_uuid: "6b6cb4d636034256bb11f126107bff9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } } I0506 07:56:45.298084 3840 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: This master's current role is: LEADER I0506 07:56:45.299284 3739 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "6b6cb4d636034256bb11f126107bff9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } } I0506 07:56:45.299749 3739 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: This master's current role is: LEADER I0506 07:56:45.300009 3841 sys_catalog.cc:335] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b6cb4d636034256bb11f126107bff9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b6cb4d636034256bb11f126107bff9f" member_type: VOTER } } I0506 07:56:45.300417 3841 sys_catalog.cc:338] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: This master's current role is: LEADER I0506 07:56:45.301312 3739 sys_catalog.cc:422] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:45.329954 3515 master_main.cc:79] Master server successfully started. W0506 07:56:45.331454 3847 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:56:45.331755 3847 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:56:45.332046 3846 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:56:45.336683 3846 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:56:45.408519 3846 catalog_manager.cc:888] Generated new certificate authority record I0506 07:56:45.410935 3846 catalog_manager.cc:1010] Loading token signing keys... I0506 07:56:45.429318 3846 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 6b6cb4d636034256bb11f126107bff9f: Generated new TSK 0 I0506 07:56:45.461462 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 3515 [ OK ] SecurityITest.SaslPlainFallback (2269 ms) [ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): setting up network... May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): skipping unrecognized local address family 17 May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): skipping unrecognized local address family 17 May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): set up 1 sockets May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): commencing operation krb5kdc: starting... W0506 07:56:45.669706 785 mini_kdc.cc:117] Time spent starting KDC: real 0.174s user 0.002s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:45 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:45.894603 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51438 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:46.205183 4047 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:46.205678 4047 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:46.206039 4047 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:46.223279 4047 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:46.223553 4047 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:46.223754 4047 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:46.223929 4047 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:46.224189 4047 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:46.224434 4047 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:46.264295 4047 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51438 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:46.309258 4047 master_main.cc:73] Initializing master server... I0506 07:56:46.312865 4047 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 198678us May 06 07:56:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:46.321432 4047 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:46.334316 4047 fs_manager.cc:260] Metadata directory not provided I0506 07:56:46.334554 4047 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:46.347718 4047 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:46.349364 4047 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:46.412130 4047 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "17ef6b3372c8459086373f9a9f723983" format_stamp: "Formatted at 2018-05-06 14:56:46 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:46.413744 4047 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "17ef6b3372c8459086373f9a9f723983" format_stamp: "Formatted at 2018-05-06 14:56:46 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:46.454100 4047 fs_manager.cc:495] Time spent creating directory manager: real 0.040s user 0.011s sys 0.007s I0506 07:56:46.456125 4047 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:46.457669 4047 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:46.473358 4047 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0506 07:56:46.473562 4047 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/wal uuid: "17ef6b3372c8459086373f9a9f723983" format_stamp: "Formatted at 2018-05-06 14:56:46 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:46.473968 4047 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:46.716778 4047 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:46.725037 4047 master_main.cc:76] Starting Master server... I0506 07:56:46.857308 4047 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:51438 I0506 07:56:46.858799 4047 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:56:46.858976 4047 webserver.cc:184] Document root disabled I0506 07:56:46.861505 4047 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:46653/ I0506 07:56:46.862573 4047 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:46.867938 4198 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:56:46.873112 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 4047 WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:56:46.897837 4198 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Bootstrap starting. I0506 07:56:46.901262 4198 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: No blocks or log segments found. Creating new log. I0506 07:56:46.903798 4198 log.cc:520] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Log is configured to *not* fsync() on all Append() calls Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:56:46.909128 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:46.916980 4198 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: No bootstrap required, opened a new log I0506 07:56:46.957032 4198 raft_consensus.cc:303] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } I0506 07:56:46.957674 4198 raft_consensus.cc:329] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:46.957924 4198 raft_consensus.cc:604] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ef6b3372c8459086373f9a9f723983, State: Initialized, Role: FOLLOWER I0506 07:56:46.958765 4198 consensus_queue.cc:227] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } I0506 07:56:46.959313 4198 raft_consensus.cc:343] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:46.959597 4198 raft_consensus.cc:435] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:46.959959 4198 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Snoozing failure detection for 2.022s (starting election) I0506 07:56:46.960228 4198 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:46.964671 4198 raft_consensus.cc:457] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } I0506 07:56:46.965322 4198 leader_election.cc:257] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:46.970376 4205 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Snoozing failure detection for 1.672s (election complete) I0506 07:56:46.971402 4205 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:46.980327 4205 raft_consensus.cc:571] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [term 1 LEADER]: Becoming Leader. State: Replica: 17ef6b3372c8459086373f9a9f723983, State: Running, Role: LEADER I0506 07:56:46.980870 4205 consensus_queue.cc:204] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } I0506 07:56:46.997365 4198 sys_catalog.cc:335] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "17ef6b3372c8459086373f9a9f723983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } } I0506 07:56:46.998306 4198 sys_catalog.cc:338] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: This master's current role is: LEADER I0506 07:56:46.998090 4206 sys_catalog.cc:335] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ef6b3372c8459086373f9a9f723983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } } I0506 07:56:47.186203 4206 sys_catalog.cc:338] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: This master's current role is: LEADER I0506 07:56:46.999069 4207 sys_catalog.cc:335] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ef6b3372c8459086373f9a9f723983. Latest consensus state: current_term: 1 leader_uuid: "17ef6b3372c8459086373f9a9f723983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ef6b3372c8459086373f9a9f723983" member_type: VOTER } } I0506 07:56:47.191800 4207 sys_catalog.cc:338] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: This master's current role is: LEADER I0506 07:56:47.198779 4198 sys_catalog.cc:422] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:47.201036 4209 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:56:47.206840 4047 master_main.cc:79] Master server successfully started. I0506 07:56:47.226474 4209 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:56:47.233839 4150 tablet.cc:1678] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Can't schedule compaction. Clean time has not been advanced past its initial value. WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:47.238867 4202 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:47.239367 4202 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:56:47.239846 4202 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:47.260547 4202 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:47.260784 4202 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:47.261358 4202 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:56:47.262017 4202 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:56:47.329876 4202 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:47.354764 4209 catalog_manager.cc:888] Generated new certificate authority record I0506 07:56:47.356508 4202 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:47.357237 4209 catalog_manager.cc:1010] Loading token signing keys... I0506 07:56:47.360234 4202 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 199178us May 06 07:56:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:47.366730 4202 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:56:47.376775 4209 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 17ef6b3372c8459086373f9a9f723983: Generated new TSK 0 I0506 07:56:47.393790 4202 fs_manager.cc:260] Metadata directory not provided I0506 07:56:47.394013 4202 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:47.394343 4202 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:56:47.394556 4202 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:47.410866 4202 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "9853093d1cf04d92a12cd2dd959829be" format_stamp: "Formatted at 2018-05-06 14:56:47 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:47.411957 4202 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "9853093d1cf04d92a12cd2dd959829be" format_stamp: "Formatted at 2018-05-06 14:56:47 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:47.437180 4202 fs_manager.cc:495] Time spent creating directory manager: real 0.025s user 0.012s sys 0.005s I0506 07:56:47.439020 4202 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:47.440660 4202 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:47.458428 4202 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s I0506 07:56:47.458556 4202 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/wal uuid: "9853093d1cf04d92a12cd2dd959829be" format_stamp: "Formatted at 2018-05-06 14:56:47 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:47.458833 4202 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:47.673410 4202 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:47.677579 4202 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:47.679597 4202 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:47.909809 4202 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:41769 I0506 07:56:47.910774 4202 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:56:47.910887 4202 webserver.cc:184] Document root disabled I0506 07:56:47.912438 4202 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:59619/ I0506 07:56:47.913161 4202 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:56:47.915174 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 4202 I0506 07:56:47.921767 4202 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. May 06 07:56:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618607, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:56:47.956917 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:47.965530 4338 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:47.916471 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51438 0506 07:56:47.926927 (+ 10456us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:47.927028 (+ 101us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:47.938482 (+ 11454us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:47.940316 (+ 1834us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:47.940346 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:47.941094 (+ 748us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:47.941363 (+ 269us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:47.942040 (+ 677us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:47.942078 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:47.943408 (+ 1330us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:47.943430 (+ 22us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:47.944857 (+ 1427us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:47.944898 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:47.948082 (+ 3184us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:47.948097 (+ 15us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:47.948447 (+ 350us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:47.948723 (+ 276us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:47.948736 (+ 13us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:47.950899 (+ 2163us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:47.957394 (+ 6495us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:47.957417 (+ 23us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:47.957438 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:47.959104 (+ 1666us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:47.960781 (+ 1677us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:47.960796 (+ 15us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:47.960805 (+ 9us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:47.960950 (+ 145us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:47.962244 (+ 1294us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:47.962260 (+ 16us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:47.962513 (+ 253us) client_negotiation.cc:695] Sending connection context 0506 07:56:47.962957 (+ 444us) client_negotiation.cc:225] Negotiation successful 0506 07:56:47.963305 (+ 348us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9941,"thread_start_us":9441,"threads_started":1} I0506 07:56:47.966095 4339 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:47.918158 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:32819 0506 07:56:47.924752 (+ 6594us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:47.924780 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:47.927195 (+ 2415us) server_negotiation.cc:373] Connection header received 0506 07:56:47.939247 (+ 12052us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:47.939283 (+ 36us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:47.939505 (+ 222us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:47.939843 (+ 338us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:47.942296 (+ 2453us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:47.943132 (+ 836us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:47.945098 (+ 1966us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:47.947342 (+ 2244us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:47.947536 (+ 194us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:47.951391 (+ 3855us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:47.951441 (+ 50us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:47.951462 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:47.951515 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:47.956743 (+ 5228us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:47.959730 (+ 2987us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:47.959758 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:47.959776 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:47.960335 (+ 559us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:47.961343 (+ 1008us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:47.961360 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:47.961370 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:47.961806 (+ 436us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:47.962162 (+ 356us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:47.963365 (+ 1203us) server_negotiation.cc:264] Negotiation successful 0506 07:56:47.963784 (+ 419us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6298,"thread_start_us":7367,"threads_started":1} I0506 07:56:47.969291 4336 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51438 I0506 07:56:47.969537 4336 heartbeater.cc:426] Registering TS with master... I0506 07:56:47.970027 4336 heartbeater.cc:475] Master 127.0.196.126:51438 requested a full tablet report, sending... I0506 07:56:47.971969 4161 ts_manager.cc:81] Registered new tserver with Master: 9853093d1cf04d92a12cd2dd959829be (127.0.196.65:41769) I0506 07:56:47.973793 4161 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:32819 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:48.144449 4342 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:48.144910 4342 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:56:48.145328 4342 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:48.161348 4342 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:48.161542 4342 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:48.161836 4342 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:56:48.162365 4342 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:56:48.193270 4342 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:48.209257 4342 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:48.211596 4342 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 199678us May 06 07:56:48 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:48.214874 4342 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:56:48.222429 4342 fs_manager.cc:260] Metadata directory not provided I0506 07:56:48.222620 4342 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:48.222961 4342 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:56:48.223165 4342 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:48.238781 4342 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "f8640b04243548bcbd814363c2d8207f" format_stamp: "Formatted at 2018-05-06 14:56:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:48.239840 4342 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "f8640b04243548bcbd814363c2d8207f" format_stamp: "Formatted at 2018-05-06 14:56:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:48.256342 4342 fs_manager.cc:495] Time spent creating directory manager: real 0.016s user 0.013s sys 0.002s I0506 07:56:48.258195 4342 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:48.258842 4342 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:48.276166 4342 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s I0506 07:56:48.276335 4342 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/wal uuid: "f8640b04243548bcbd814363c2d8207f" format_stamp: "Formatted at 2018-05-06 14:56:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:48.276644 4342 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:48.483242 4342 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:48.492866 4342 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:48.501806 4342 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:48.930213 4342 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:57266 I0506 07:56:48.931504 4342 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:56:48.931632 4342 webserver.cc:184] Document root disabled I0506 07:56:48.934358 4342 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:42631/ I0506 07:56:48.935303 4342 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:56:48.941260 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 4342 I0506 07:56:48.948096 4342 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. May 06 07:56:48 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:48.973978 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:48.976613 4336 heartbeater.cc:467] Master 127.0.196.126:51438 was elected leader, sending a full tablet report... I0506 07:56:48.993595 4510 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:48.940952 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:59391 0506 07:56:48.944449 (+ 3497us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:48.944474 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:48.954957 (+ 10483us) server_negotiation.cc:373] Connection header received 0506 07:56:48.963862 (+ 8905us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:48.963891 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:48.963983 (+ 92us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:48.964132 (+ 149us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:48.966737 (+ 2605us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:48.967352 (+ 615us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:48.969073 (+ 1721us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:48.970009 (+ 936us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:48.970170 (+ 161us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:48.974384 (+ 4214us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:48.974449 (+ 65us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:48.974478 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:48.974563 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:48.981715 (+ 7152us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:48.984536 (+ 2821us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:48.984562 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:48.984578 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:48.985222 (+ 644us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:48.986803 (+ 1581us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:48.986826 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:48.986841 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:48.987287 (+ 446us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:48.987620 (+ 333us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:48.991098 (+ 3478us) server_negotiation.cc:264] Negotiation successful 0506 07:56:48.991398 (+ 300us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3232,"thread_start_us":2532,"threads_started":1} I0506 07:56:48.996456 4511 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:48.942322 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51438 0506 07:56:48.951436 (+ 9114us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:48.951519 (+ 83us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:48.963540 (+ 12021us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:48.964410 (+ 870us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:48.964444 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:48.965293 (+ 849us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:48.965586 (+ 293us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:48.966479 (+ 893us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:48.966523 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:48.967642 (+ 1119us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:48.967659 (+ 17us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:48.968884 (+ 1225us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:48.968918 (+ 34us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:48.970240 (+ 1322us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:48.970269 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:48.970645 (+ 376us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:48.970874 (+ 229us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:48.970890 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:48.973600 (+ 2710us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:48.982456 (+ 8856us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:48.982486 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:48.982510 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:48.983892 (+ 1382us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:48.985860 (+ 1968us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:48.985885 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:48.985905 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:48.986160 (+ 255us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:48.988054 (+ 1894us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:48.988083 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:48.988533 (+ 450us) client_negotiation.cc:695] Sending connection context 0506 07:56:48.989115 (+ 582us) client_negotiation.cc:225] Negotiation successful 0506 07:56:48.989488 (+ 373us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":8625,"thread_start_us":11500,"threads_started":1} I0506 07:56:48.999577 4508 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51438 I0506 07:56:48.999920 4508 heartbeater.cc:426] Registering TS with master... I0506 07:56:49.000603 4508 heartbeater.cc:475] Master 127.0.196.126:51438 requested a full tablet report, sending... I0506 07:56:49.002679 4161 ts_manager.cc:81] Registered new tserver with Master: f8640b04243548bcbd814363c2d8207f (127.0.196.66:57266) I0506 07:56:49.004716 4161 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:59391 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:49.170819 4514 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:49.171277 4514 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:56:49.171702 4514 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:49.186283 4514 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:49.186555 4514 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:49.186873 4514 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:56:49.187384 4514 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:56:49.243995 4514 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51438 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:49.259966 4514 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:49.262845 4514 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 200178us May 06 07:56:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:49.270349 4514 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:56:49.285112 4514 fs_manager.cc:260] Metadata directory not provided I0506 07:56:49.285364 4514 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:49.285775 4514 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:56:49.286020 4514 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:49.301270 4514 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "6ed94894949348d9863a4061102f924a" format_stamp: "Formatted at 2018-05-06 14:56:49 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:49.302342 4514 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "6ed94894949348d9863a4061102f924a" format_stamp: "Formatted at 2018-05-06 14:56:49 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:49.341928 4514 fs_manager.cc:495] Time spent creating directory manager: real 0.039s user 0.013s sys 0.004s I0506 07:56:49.343806 4514 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:49.345207 4514 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:49.358863 4514 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0506 07:56:49.359061 4514 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/wal uuid: "6ed94894949348d9863a4061102f924a" format_stamp: "Formatted at 2018-05-06 14:56:49 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:49.359521 4514 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:49.542057 4514 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:49.545261 4514 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:49.546528 4514 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:49.701367 4514 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:37533 I0506 07:56:49.702291 4514 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:56:49.702401 4514 webserver.cc:184] Document root disabled I0506 07:56:49.703986 4514 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:36471/ I0506 07:56:49.704676 4514 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:56:49.705406 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 4514 I0506 07:56:49.717617 4514 tablet_server_main.cc:87] Tablet server successfully started. May 06 07:56:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:56:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618609, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:49.751603 4690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:49.727135 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:39875 0506 07:56:49.728433 (+ 1298us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:49.728458 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:49.728491 (+ 33us) server_negotiation.cc:373] Connection header received 0506 07:56:49.728612 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:49.728636 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:49.728753 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:49.728907 (+ 154us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:49.730642 (+ 1735us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.731281 (+ 639us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.732660 (+ 1379us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.734200 (+ 1540us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.734391 (+ 191us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:49.737592 (+ 3201us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:49.737621 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:49.737647 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:49.737747 (+ 100us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:49.744899 (+ 7152us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.746334 (+ 1435us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.746360 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.746373 (+ 13us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.746824 (+ 451us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.747264 (+ 440us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.747291 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.747313 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.747896 (+ 583us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:49.748259 (+ 363us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:49.749284 (+ 1025us) server_negotiation.cc:264] Negotiation successful 0506 07:56:49.749771 (+ 487us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1107,"thread_start_us":858,"threads_started":1} I0506 07:56:49.765909 4685 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:49.711366 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:46075 0506 07:56:49.727300 (+ 15934us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:49.727325 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:49.727362 (+ 37us) server_negotiation.cc:373] Connection header received 0506 07:56:49.740703 (+ 13341us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:49.740730 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:49.740854 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:49.741029 (+ 175us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:49.743078 (+ 2049us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.743674 (+ 596us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.745655 (+ 1981us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.747137 (+ 1482us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.747306 (+ 169us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:49.753030 (+ 5724us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:49.753086 (+ 56us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:49.753112 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:49.753192 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:49.757941 (+ 4749us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.759763 (+ 1822us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.759793 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.759811 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.760419 (+ 608us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.761760 (+ 1341us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.761777 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.761788 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.762196 (+ 408us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:49.762555 (+ 359us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:49.763743 (+ 1188us) server_negotiation.cc:264] Negotiation successful 0506 07:56:49.764062 (+ 319us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15683,"thread_start_us":15447,"threads_started":1} I0506 07:56:49.766180 4687 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:49.712532 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51438 0506 07:56:49.721087 (+ 8555us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:49.721195 (+ 108us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:49.740408 (+ 19213us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:49.741255 (+ 847us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:49.741291 (+ 36us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:49.742109 (+ 818us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:49.742396 (+ 287us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:49.742841 (+ 445us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:49.742889 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.743864 (+ 975us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.743892 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:49.745435 (+ 1543us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:49.745479 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.747367 (+ 1888us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.747396 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:49.747877 (+ 481us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:49.748188 (+ 311us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:49.748210 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:49.752472 (+ 4262us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:49.758409 (+ 5937us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:49.758427 (+ 18us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:49.758442 (+ 15us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:49.759277 (+ 835us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:49.761058 (+ 1781us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:49.761077 (+ 19us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:49.761087 (+ 10us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:49.761274 (+ 187us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:49.762685 (+ 1411us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:49.762712 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:49.763014 (+ 302us) client_negotiation.cc:695] Sending connection context 0506 07:56:49.763417 (+ 403us) client_negotiation.cc:225] Negotiation successful 0506 07:56:49.763804 (+ 387us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":8148,"thread_start_us":7673,"threads_started":1} I0506 07:56:49.768914 4679 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51438 I0506 07:56:49.769232 4679 heartbeater.cc:426] Registering TS with master... I0506 07:56:49.769840 4679 heartbeater.cc:475] Master 127.0.196.126:51438 requested a full tablet report, sending... I0506 07:56:49.771865 4161 ts_manager.cc:81] Registered new tserver with Master: 6ed94894949348d9863a4061102f924a (127.0.196.67:37533) I0506 07:56:49.773843 4161 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:46075 I0506 07:56:49.774230 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters May 06 07:56:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for joe-interloper@KRBTEST.COM: May 06 07:56:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618609, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:49.822453 4685 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:49.803452 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:39876 0506 07:56:49.803724 (+ 272us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:49.803741 (+ 17us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:49.805902 (+ 2161us) server_negotiation.cc:373] Connection header received 0506 07:56:49.806140 (+ 238us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:49.806157 (+ 17us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:49.806245 (+ 88us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:49.806354 (+ 109us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:49.807870 (+ 1516us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.808239 (+ 369us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.809545 (+ 1306us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:49.810431 (+ 886us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:49.810594 (+ 163us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:49.813493 (+ 2899us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:49.813511 (+ 18us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:49.813524 (+ 13us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:49.813570 (+ 46us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:49.817426 (+ 3856us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.818554 (+ 1128us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.818572 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.818583 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.819013 (+ 430us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:49.819400 (+ 387us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:49.819417 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:49.819427 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:49.819772 (+ 345us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:49.820042 (+ 270us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:49.820529 (+ 487us) server_negotiation.cc:264] Negotiation successful 0506 07:56:49.820997 (+ 468us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":87} W0506 07:56:49.823187 4161 server_base.cc:567] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.196.126:39876 I0506 07:56:49.829396 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 4202 W0506 07:56:49.853335 4142 connection.cc:512] server connection from 127.0.196.65:32819 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:32819): Connection reset by peer (error 104) I0506 07:56:49.854367 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 4342 I0506 07:56:49.899025 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 4514 W0506 07:56:49.909138 4145 connection.cc:512] server connection from 127.0.196.66:59391 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:59391): Connection reset by peer (error 104) I0506 07:56:49.930953 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 4047 [ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4474 ms) [ RUN ] SecurityITest.TestAuthorizedSuperuser Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): setting up network... May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): skipping unrecognized local address family 17 May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): skipping unrecognized local address family 17 May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): set up 1 sockets May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): commencing operation krb5kdc: starting... W0506 07:56:50.076244 785 mini_kdc.cc:117] Time spent starting KDC: real 0.107s user 0.004s sys 0.009s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:50.239193 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:43753 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:50.436918 4792 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:50.437366 4792 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:50.437757 4792 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:50.453717 4792 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:50.453930 4792 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:50.454111 4792 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:50.454291 4792 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:50.454501 4792 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:50.454658 4792 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:50.508743 4792 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:43753 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:50.523442 4792 master_main.cc:73] Initializing master server... I0506 07:56:50.526893 4792 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 200678us May 06 07:56:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:50.533287 4792 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:50.546761 4792 fs_manager.cc:260] Metadata directory not provided I0506 07:56:50.546979 4792 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:50.547305 4792 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:50.547508 4792 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:50.563315 4792 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "fcc745e100f2448e95b7a1f120dea53e" format_stamp: "Formatted at 2018-05-06 14:56:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:50.564352 4792 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "fcc745e100f2448e95b7a1f120dea53e" format_stamp: "Formatted at 2018-05-06 14:56:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:50.593672 4792 fs_manager.cc:495] Time spent creating directory manager: real 0.029s user 0.013s sys 0.014s I0506 07:56:50.595377 4792 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:50.596843 4792 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:50.610636 4792 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0506 07:56:50.610842 4792 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/wal uuid: "fcc745e100f2448e95b7a1f120dea53e" format_stamp: "Formatted at 2018-05-06 14:56:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:50.611083 4792 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:50.870579 4792 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:50.879937 4792 master_main.cc:76] Starting Master server... I0506 07:56:51.069896 4792 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:43753 I0506 07:56:51.071235 4792 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:56:51.071408 4792 webserver.cc:184] Document root disabled I0506 07:56:51.074039 4792 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:38105/ I0506 07:56:51.075131 4792 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:51.080389 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 4792 I0506 07:56:51.091526 5024 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:56:51.126227 5024 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: Bootstrap starting. I0506 07:56:51.129328 5024 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: No blocks or log segments found. Creating new log. I0506 07:56:51.137786 5024 log.cc:520] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: Log is configured to *not* fsync() on all Append() calls Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:56:51.143075 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:51.149011 5024 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: No bootstrap required, opened a new log I0506 07:56:51.174046 5024 raft_consensus.cc:303] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } I0506 07:56:51.174513 5024 raft_consensus.cc:329] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:51.174746 5024 raft_consensus.cc:604] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc745e100f2448e95b7a1f120dea53e, State: Initialized, Role: FOLLOWER I0506 07:56:51.181749 5024 consensus_queue.cc:227] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } I0506 07:56:51.182315 5024 raft_consensus.cc:343] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:51.182564 5024 raft_consensus.cc:435] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:51.182831 5024 raft_consensus.cc:2704] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: Snoozing failure detection for 2.084s (starting election) I0506 07:56:51.183069 5024 raft_consensus.cc:2756] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:51.194231 5024 raft_consensus.cc:457] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } I0506 07:56:51.194947 5024 leader_election.cc:257] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:51.201393 5049 raft_consensus.cc:2704] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: Snoozing failure detection for 1.924s (election complete) I0506 07:56:51.201704 5049 raft_consensus.cc:2511] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:51.213498 5049 raft_consensus.cc:571] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [term 1 LEADER]: Becoming Leader. State: Replica: fcc745e100f2448e95b7a1f120dea53e, State: Running, Role: LEADER I0506 07:56:51.222306 5049 consensus_queue.cc:204] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } I0506 07:56:51.232586 5052 sys_catalog.cc:335] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcc745e100f2448e95b7a1f120dea53e. Latest consensus state: current_term: 1 leader_uuid: "fcc745e100f2448e95b7a1f120dea53e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } } I0506 07:56:51.233170 5052 sys_catalog.cc:338] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: This master's current role is: LEADER WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:51.381644 5045 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:51.382220 5045 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:56:51.382650 5045 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:51.399060 5045 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:51.399268 5045 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:51.399541 5045 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:56:51.400074 5045 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:56:51.234706 5051 sys_catalog.cc:335] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcc745e100f2448e95b7a1f120dea53e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } } I0506 07:56:51.422492 5051 sys_catalog.cc:338] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: This master's current role is: LEADER I0506 07:56:51.234874 5024 sys_catalog.cc:335] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "fcc745e100f2448e95b7a1f120dea53e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc745e100f2448e95b7a1f120dea53e" member_type: VOTER } } I0506 07:56:51.431839 5024 sys_catalog.cc:338] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: This master's current role is: LEADER I0506 07:56:51.433320 5024 sys_catalog.cc:422] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:51.438541 5045 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:51.450789 4792 master_main.cc:79] Master server successfully started. I0506 07:56:51.454556 5045 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:51.458123 5045 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 201178us W0506 07:56:51.458878 5080 catalog_manager.cc:1036] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:56:51.459110 5080 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:56:51.459400 5055 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:56:51.462715 5055 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... May 06 07:56:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:51.464833 5045 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:56:51.478026 5045 fs_manager.cc:260] Metadata directory not provided I0506 07:56:51.478246 5045 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:51.478600 5045 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:56:51.478866 5045 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:51.494870 5045 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "3c23621469fd47b5936450eba896e895" format_stamp: "Formatted at 2018-05-06 14:56:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:51.495935 5045 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "3c23621469fd47b5936450eba896e895" format_stamp: "Formatted at 2018-05-06 14:56:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:51.517832 5045 fs_manager.cc:495] Time spent creating directory manager: real 0.021s user 0.013s sys 0.004s I0506 07:56:51.519604 5045 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:51.521029 5045 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:51.546097 5045 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.000s sys 0.007s I0506 07:56:51.546239 5045 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/wal uuid: "3c23621469fd47b5936450eba896e895" format_stamp: "Formatted at 2018-05-06 14:56:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:51.546483 5045 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:51.593013 5055 catalog_manager.cc:888] Generated new certificate authority record I0506 07:56:51.595813 5055 catalog_manager.cc:1010] Loading token signing keys... I0506 07:56:51.652240 5055 catalog_manager.cc:3992] T 00000000000000000000000000000000 P fcc745e100f2448e95b7a1f120dea53e: Generated new TSK 0 I0506 07:56:52.215327 5045 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:52.223883 5045 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:52.230417 5045 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:53.328253 5045 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:45141 I0506 07:56:53.329743 5045 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:56:53.329948 5045 webserver.cc:184] Document root disabled I0506 07:56:53.335873 5045 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:39854/ I0506 07:56:53.336835 5045 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:56:53.342732 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 5045 I0506 07:56:53.374542 5045 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. May 06 07:56:53 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618611, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:53.435547 5380 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:53.345051 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:43753 0506 07:56:53.361576 (+ 16525us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:53.361679 (+ 103us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:53.391095 (+ 29416us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:53.392557 (+ 1462us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:53.392615 (+ 58us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:53.393240 (+ 625us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:53.393437 (+ 197us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:53.394064 (+ 627us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:53.394092 (+ 28us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:53.395352 (+ 1260us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:53.395376 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:53.396816 (+ 1440us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:53.396855 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:53.401739 (+ 4884us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:53.401755 (+ 16us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:53.402044 (+ 289us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:53.402289 (+ 245us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:53.402302 (+ 13us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:53.405483 (+ 3181us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:53.425113 (+ 19630us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:53.425141 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:53.425165 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:53.426383 (+ 1218us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:53.428617 (+ 2234us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:53.428644 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:53.428663 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:53.428880 (+ 217us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:53.431043 (+ 2163us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:53.431075 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:53.431483 (+ 408us) client_negotiation.cc:695] Sending connection context 0506 07:56:53.432041 (+ 558us) client_negotiation.cc:225] Negotiation successful 0506 07:56:53.432399 (+ 358us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":16155,"thread_start_us":26665,"threads_started":1} I0506 07:56:53.446054 5381 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:53.346220 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:42524 0506 07:56:53.355186 (+ 8966us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:53.355226 (+ 40us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:53.371717 (+ 16491us) server_negotiation.cc:373] Connection header received 0506 07:56:53.391773 (+ 20056us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:53.391814 (+ 41us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:53.392055 (+ 241us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:53.392408 (+ 353us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:53.394267 (+ 1859us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:53.395156 (+ 889us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:53.397216 (+ 2060us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:53.399525 (+ 2309us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:53.399751 (+ 226us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:53.406325 (+ 6574us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:53.406408 (+ 83us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:53.406442 (+ 34us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:53.406527 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:53.424400 (+ 17873us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:53.427196 (+ 2796us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:53.427226 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:53.427246 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:53.427943 (+ 697us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:53.429621 (+ 1678us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:53.429648 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:53.429667 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:53.430332 (+ 665us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:53.437570 (+ 7238us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:53.439089 (+ 1519us) server_negotiation.cc:264] Negotiation successful 0506 07:56:53.439749 (+ 660us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8609,"thread_start_us":8555,"threads_started":1} I0506 07:56:53.450064 5378 heartbeater.cc:346] Connected to a master server at 127.0.196.126:43753 I0506 07:56:53.450372 5378 heartbeater.cc:426] Registering TS with master... I0506 07:56:53.451027 5378 heartbeater.cc:475] Master 127.0.196.126:43753 requested a full tablet report, sending... I0506 07:56:53.453786 4963 ts_manager.cc:81] Registered new tserver with Master: 3c23621469fd47b5936450eba896e895 (127.0.196.65:45141) Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. W0506 07:56:53.454784 785 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.196.66: real 0.111s user 0.001s sys 0.009s I0506 07:56:53.456261 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:53.457231 4963 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:42524 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:53.782189 5386 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:53.786106 5386 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:56:53.786849 5386 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:53.811811 5386 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:53.812259 5386 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:53.813915 5386 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:56:53.814746 5386 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:56:53.884737 5386 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:53.901552 5386 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:53.904079 5386 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 202178us May 06 07:56:53 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:53.907665 5386 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:56:53.938015 5386 fs_manager.cc:260] Metadata directory not provided I0506 07:56:53.938262 5386 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:53.938649 5386 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:56:53.938899 5386 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:53.965754 5386 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "d85ff4fa5c824d35ab9401225de2892a" format_stamp: "Formatted at 2018-05-06 14:56:53 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:53.966919 5386 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "d85ff4fa5c824d35ab9401225de2892a" format_stamp: "Formatted at 2018-05-06 14:56:53 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:53.999421 5386 fs_manager.cc:495] Time spent creating directory manager: real 0.032s user 0.015s sys 0.005s I0506 07:56:54.002001 5386 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:54.003500 5386 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:54.021448 5386 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s I0506 07:56:54.025522 5386 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/wal uuid: "d85ff4fa5c824d35ab9401225de2892a" format_stamp: "Formatted at 2018-05-06 14:56:53 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:54.025954 5386 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0506 07:56:54.122962 5163 debug-util.cc:397] Leaking SignalData structure 0x7b0800029080 after lost signal to thread 5073 I0506 07:56:54.369829 5386 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:54.381436 5386 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:54.392529 5386 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:54.463054 5378 heartbeater.cc:467] Master 127.0.196.126:43753 was elected leader, sending a full tablet report... I0506 07:56:55.328783 5386 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:47533 I0506 07:56:55.330492 5386 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:56:55.330680 5386 webserver.cc:184] Document root disabled I0506 07:56:55.357828 5386 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:51087/ I0506 07:56:55.385723 5386 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:56:55.394385 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 5386 I0506 07:56:55.410156 5386 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. I0506 07:56:55.460412 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:56:55 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:55.508937 5685 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:55.398059 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:34264 0506 07:56:55.406884 (+ 8825us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:55.406903 (+ 19us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:55.408480 (+ 1577us) server_negotiation.cc:373] Connection header received 0506 07:56:55.445297 (+ 36817us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:55.445329 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:55.445454 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:55.445591 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:55.471449 (+ 25858us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:55.472037 (+ 588us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:55.474247 (+ 2210us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:55.475608 (+ 1361us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:55.475806 (+ 198us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:55.488750 (+ 12944us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:55.488818 (+ 68us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:55.488841 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:55.488919 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:55.495559 (+ 6640us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:55.499291 (+ 3732us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:55.499315 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:55.499330 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:55.500032 (+ 702us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:55.501715 (+ 1683us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:55.501738 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:55.501753 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:55.502275 (+ 522us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:55.502653 (+ 378us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:55.506934 (+ 4281us) server_negotiation.cc:264] Negotiation successful 0506 07:56:55.507229 (+ 295us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8633,"thread_start_us":8238,"threads_started":1} I0506 07:56:55.515658 5687 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:55.400349 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:43753 0506 07:56:55.408151 (+ 7802us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:55.408293 (+ 142us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:55.444157 (+ 35864us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:55.446189 (+ 2032us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:55.446221 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:55.447075 (+ 854us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:55.448488 (+ 1413us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:55.449740 (+ 1252us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:55.450342 (+ 602us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:55.472343 (+ 22001us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:55.472368 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:55.473969 (+ 1601us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:55.474019 (+ 50us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:55.476309 (+ 2290us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:55.476340 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:55.476779 (+ 439us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:55.477098 (+ 319us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:55.477117 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:55.483469 (+ 6352us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:55.496351 (+ 12882us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:55.496383 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:55.496407 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:55.498651 (+ 2244us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:55.500657 (+ 2006us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:55.500684 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:55.500703 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:55.500985 (+ 282us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:55.503049 (+ 2064us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:55.503076 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:55.503646 (+ 570us) client_negotiation.cc:695] Sending connection context 0506 07:56:55.504243 (+ 597us) client_negotiation.cc:225] Negotiation successful 0506 07:56:55.504661 (+ 418us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6975,"thread_start_us":4337,"threads_started":1} I0506 07:56:55.518581 5683 heartbeater.cc:346] Connected to a master server at 127.0.196.126:43753 I0506 07:56:55.518934 5683 heartbeater.cc:426] Registering TS with master... I0506 07:56:55.519567 5683 heartbeater.cc:475] Master 127.0.196.126:43753 requested a full tablet report, sending... I0506 07:56:55.521823 4963 ts_manager.cc:81] Registered new tserver with Master: d85ff4fa5c824d35ab9401225de2892a (127.0.196.66:47533) I0506 07:56:55.523739 4963 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:34264 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:55.741942 5694 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:55.742249 5694 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:56:55.742496 5694 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:55.754511 5694 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:55.754650 5694 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:55.754837 5694 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:56:55.755167 5694 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:56:55.788604 5694 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:43753 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:55.802227 5694 tablet_server_main.cc:79] Initializing tablet server... I0506 07:56:55.805963 5694 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 203178us May 06 07:56:55 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:55.813158 5694 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:56:55.826599 5694 fs_manager.cc:260] Metadata directory not provided I0506 07:56:55.826875 5694 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:55.827229 5694 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:56:55.827481 5694 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:55.840256 5694 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "07c848c6605940cba7f33fe96320aeb5" format_stamp: "Formatted at 2018-05-06 14:56:55 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:55.841445 5694 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "07c848c6605940cba7f33fe96320aeb5" format_stamp: "Formatted at 2018-05-06 14:56:55 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:55.859827 5694 fs_manager.cc:495] Time spent creating directory manager: real 0.018s user 0.013s sys 0.002s I0506 07:56:55.861995 5694 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:55.864758 5694 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:55.907927 5694 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.001s I0506 07:56:55.908133 5694 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/wal uuid: "07c848c6605940cba7f33fe96320aeb5" format_stamp: "Formatted at 2018-05-06 14:56:55 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:55.908504 5694 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:56.266873 5694 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:56.282286 5694 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:56:56.289161 5694 tablet_server_main.cc:84] Starting tablet server... I0506 07:56:56.527777 5683 heartbeater.cc:467] Master 127.0.196.126:43753 was elected leader, sending a full tablet report... I0506 07:56:57.045398 5694 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:42744 I0506 07:56:57.046845 5694 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:56:57.047114 5694 webserver.cc:184] Document root disabled I0506 07:56:57.061156 5694 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:32973/ I0506 07:56:57.062121 5694 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:56:57.064093 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 5694 I0506 07:56:57.089032 5694 tablet_server_main.cc:87] Tablet server successfully started. May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:57.124511 6073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:57.065294 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34526 0506 07:56:57.076459 (+ 11165us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:57.076487 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:57.080121 (+ 3634us) server_negotiation.cc:373] Connection header received 0506 07:56:57.080242 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:57.080268 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:57.080392 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:57.080532 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:57.086756 (+ 6224us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.087352 (+ 596us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.094761 (+ 7409us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.096283 (+ 1522us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.096458 (+ 175us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:57.108598 (+ 12140us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:57.108629 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:57.108677 (+ 48us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:57.108761 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:57.115506 (+ 6745us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.117334 (+ 1828us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.117362 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.117381 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.118071 (+ 690us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.118699 (+ 628us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.118727 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.118747 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.119257 (+ 510us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:57.119619 (+ 362us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:57.121524 (+ 1905us) server_negotiation.cc:264] Negotiation successful 0506 07:56:57.122238 (+ 714us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10953,"thread_start_us":10752,"threads_started":1} May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618615, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:57.184365 6077 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:57.081557 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:50320 0506 07:56:57.083200 (+ 1643us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:57.083226 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:57.094159 (+ 10933us) server_negotiation.cc:373] Connection header received 0506 07:56:57.104780 (+ 10621us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:57.104807 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:57.104916 (+ 109us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:57.105053 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:57.112097 (+ 7044us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.112721 (+ 624us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.127585 (+ 14864us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.129169 (+ 1584us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.129351 (+ 182us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:57.149087 (+ 19736us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:57.149159 (+ 72us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:57.149187 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:57.149292 (+ 105us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:57.163684 (+ 14392us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.168546 (+ 4862us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.168571 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.168590 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.169273 (+ 683us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.170402 (+ 1129us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.170419 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.170458 (+ 39us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.173075 (+ 2617us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:57.173418 (+ 343us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:57.182183 (+ 8765us) server_negotiation.cc:264] Negotiation successful 0506 07:56:57.182517 (+ 334us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":5025,"threads_started":1} I0506 07:56:57.189812 6076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:57.079820 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:43753 0506 07:56:57.089542 (+ 9722us) negotiation.cc:98] Waiting for socket to connect 0506 07:56:57.089616 (+ 74us) client_negotiation.cc:160] Beginning negotiation 0506 07:56:57.104489 (+ 14873us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:56:57.105557 (+ 1068us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:56:57.105590 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:56:57.106357 (+ 767us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:56:57.106663 (+ 306us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:57.106983 (+ 320us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:57.107029 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.124758 (+ 17729us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.124782 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:57.126777 (+ 1995us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:56:57.127367 (+ 590us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.136728 (+ 9361us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.136758 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:56:57.137385 (+ 627us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:57.140313 (+ 2928us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:56:57.141018 (+ 705us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:56:57.146441 (+ 5423us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:56:57.166695 (+ 20254us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:57.166725 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:57.166744 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:57.167583 (+ 839us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:57.169804 (+ 2221us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:56:57.169822 (+ 18us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:56:57.169834 (+ 12us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:56:57.169994 (+ 160us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:56:57.173542 (+ 3548us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:56:57.173575 (+ 33us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:56:57.174068 (+ 493us) client_negotiation.cc:695] Sending connection context 0506 07:56:57.177004 (+ 2936us) client_negotiation.cc:225] Negotiation successful 0506 07:56:57.177386 (+ 382us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9405,"thread_start_us":3292,"threads_started":1} I0506 07:56:57.194023 6071 heartbeater.cc:346] Connected to a master server at 127.0.196.126:43753 I0506 07:56:57.194358 6071 heartbeater.cc:426] Registering TS with master... I0506 07:56:57.194963 6071 heartbeater.cc:475] Master 127.0.196.126:43753 requested a full tablet report, sending... I0506 07:56:57.197253 4963 ts_manager.cc:81] Registered new tserver with Master: 07c848c6605940cba7f33fe96320aeb5 (127.0.196.67:42744) I0506 07:56:57.198601 4963 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:50320 I0506 07:56:57.203021 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.65@KRBTEST.COM I0506 07:56:57.302307 6092 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:57.262717 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:37309 0506 07:56:57.274584 (+ 11867us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:57.274642 (+ 58us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:57.274693 (+ 51us) server_negotiation.cc:373] Connection header received 0506 07:56:57.274810 (+ 117us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:57.274837 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:57.275014 (+ 177us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:57.275174 (+ 160us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:57.282102 (+ 6928us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.282731 (+ 629us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.284698 (+ 1967us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.286080 (+ 1382us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.286242 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:57.290454 (+ 4212us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:57.290482 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:57.290499 (+ 17us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:57.290549 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:57.294716 (+ 4167us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.296287 (+ 1571us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.296318 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.296339 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.296985 (+ 646us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.297541 (+ 556us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.297567 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.297585 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.298126 (+ 541us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:57.298483 (+ 357us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:57.299956 (+ 1473us) server_negotiation.cc:264] Negotiation successful 0506 07:56:57.300396 (+ 440us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11647,"thread_start_us":5409,"threads_started":1} May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[4745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:56:57.345104 6077 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:56:57.315381 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34534 0506 07:56:57.315685 (+ 304us) server_negotiation.cc:176] Beginning negotiation 0506 07:56:57.315730 (+ 45us) server_negotiation.cc:365] Waiting for connection header 0506 07:56:57.316983 (+ 1253us) server_negotiation.cc:373] Connection header received 0506 07:56:57.317336 (+ 353us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:56:57.317362 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:56:57.317482 (+ 120us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:56:57.317612 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:56:57.319802 (+ 2190us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.320352 (+ 550us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.322410 (+ 2058us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:56:57.323867 (+ 1457us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:56:57.324029 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:56:57.330591 (+ 6562us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:56:57.330619 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:56:57.330642 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:56:57.330735 (+ 93us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:56:57.336851 (+ 6116us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.338618 (+ 1767us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.338653 (+ 35us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.338694 (+ 41us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.339302 (+ 608us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:56:57.339940 (+ 638us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:56:57.339966 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:56:57.339985 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:56:57.340437 (+ 452us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:56:57.340744 (+ 307us) server_negotiation.cc:901] Waiting for connection context 0506 07:56:57.342555 (+ 1811us) server_negotiation.cc:264] Negotiation successful 0506 07:56:57.343002 (+ 447us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":99} W0506 07:56:57.345865 4963 server_base.cc:567] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.196.126:34534 I0506 07:56:57.348399 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 5045 W0506 07:56:57.390245 4888 connection.cc:512] server connection from 127.0.196.65:42524 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:42524): Connection reset by peer (error 104) I0506 07:56:57.391168 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 5386 W0506 07:56:57.420912 4884 connection.cc:512] server connection from 127.0.196.66:34264 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:34264): Connection reset by peer (error 104) I0506 07:56:57.421870 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 5694 W0506 07:56:57.457120 4884 connection.cc:512] server connection from 127.0.196.67:50320 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:50320): Connection reset by peer (error 104) I0506 07:56:57.457173 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 4792 [ OK ] SecurityITest.TestAuthorizedSuperuser (7542 ms) [ RUN ] SecurityITest.TestDisableWebUI Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): setting up network... May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): skipping unrecognized local address family 17 May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): skipping unrecognized local address family 17 May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): set up 1 sockets May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): commencing operation krb5kdc: starting... W0506 07:56:57.678205 785 mini_kdc.cc:117] Time spent starting KDC: real 0.165s user 0.003s sys 0.013s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:56:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:56:57.945832 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51336 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:58.297036 6186 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:58.297515 6186 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:56:58.298036 6186 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:58.323065 6186 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:56:58.323276 6186 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:58.323457 6186 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:58.323668 6186 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:56:58.323884 6186 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:56:58.324113 6186 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:56:58.398672 6186 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51336 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:56:58.421079 6186 master_main.cc:73] Initializing master server... I0506 07:56:58.424208 6186 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 204678us May 06 07:56:58 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:56:58.439046 6186 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:56:58.459748 6186 fs_manager.cc:260] Metadata directory not provided I0506 07:56:58.459945 6186 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:56:58.460265 6186 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:56:58.460454 6186 server_base.cc:434] Attempting to create new FS layout instead I0506 07:56:58.484998 6186 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "3b63a104e88347dbb42b826d60c0c1e5" format_stamp: "Formatted at 2018-05-06 14:56:58 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:58.486156 6186 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "3b63a104e88347dbb42b826d60c0c1e5" format_stamp: "Formatted at 2018-05-06 14:56:58 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:58.514582 6186 fs_manager.cc:495] Time spent creating directory manager: real 0.028s user 0.011s sys 0.013s I0506 07:56:58.516502 6186 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:56:58.517990 6186 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:56:58.535816 6186 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s I0506 07:56:58.536026 6186 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/wal uuid: "3b63a104e88347dbb42b826d60c0c1e5" format_stamp: "Formatted at 2018-05-06 14:56:58 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:56:58.536412 6186 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:56:58.878422 6186 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:56:58.885721 6186 master_main.cc:76] Starting Master server... I0506 07:56:59.243855 6186 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:51336 I0506 07:56:59.244884 6186 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:56:59.245642 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 6186 I0506 07:56:59.261770 6467 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:56:59.294294 6467 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Bootstrap starting. I0506 07:56:59.297763 6467 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: No blocks or log segments found. Creating new log. I0506 07:56:59.300941 6467 log.cc:520] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Log is configured to *not* fsync() on all Append() calls I0506 07:56:59.316506 6467 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:56:59.339979 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:56:59.340874 6467 raft_consensus.cc:303] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } I0506 07:56:59.341835 6467 raft_consensus.cc:329] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:56:59.342084 6467 raft_consensus.cc:604] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b63a104e88347dbb42b826d60c0c1e5, State: Initialized, Role: FOLLOWER I0506 07:56:59.342985 6467 consensus_queue.cc:227] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } I0506 07:56:59.343518 6467 raft_consensus.cc:343] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:56:59.343827 6467 raft_consensus.cc:435] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:56:59.344488 6467 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Snoozing failure detection for 2.039s (starting election) I0506 07:56:59.344750 6467 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:56:59.349957 6467 raft_consensus.cc:457] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } I0506 07:56:59.350551 6467 leader_election.cc:257] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:56:59.356031 6488 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Snoozing failure detection for 1.588s (election complete) I0506 07:56:59.359714 6488 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:56:59.382295 6488 raft_consensus.cc:571] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [term 1 LEADER]: Becoming Leader. State: Replica: 3b63a104e88347dbb42b826d60c0c1e5, State: Running, Role: LEADER I0506 07:56:59.382905 6488 consensus_queue.cc:204] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } I0506 07:56:59.396652 6467 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "3b63a104e88347dbb42b826d60c0c1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } } I0506 07:56:59.397101 6467 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: This master's current role is: LEADER I0506 07:56:59.398706 6491 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b63a104e88347dbb42b826d60c0c1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } } I0506 07:56:59.609194 6491 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: This master's current role is: LEADER I0506 07:56:59.399022 6492 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b63a104e88347dbb42b826d60c0c1e5. Latest consensus state: current_term: 1 leader_uuid: "3b63a104e88347dbb42b826d60c0c1e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b63a104e88347dbb42b826d60c0c1e5" member_type: VOTER } } I0506 07:56:59.622169 6492 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: This master's current role is: LEADER I0506 07:56:59.623451 6467 sys_catalog.cc:422] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:56:59.623806 6498 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:56:59.634959 6186 master_main.cc:79] Master server successfully started. I0506 07:56:59.643815 6498 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:56:59.649314 6352 tablet.cc:1678] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Can't schedule compaction. Clean time has not been advanced past its initial value. WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:56:59.880251 6486 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:56:59.891831 6486 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:56:59.898650 6486 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:56:59.955334 6486 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:56:59.957057 6486 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:56:59.957546 6486 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:56:59.958793 6486 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:57:00.007207 6498 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:00.009785 6498 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:00.081064 6498 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Generated new TSK 0 I0506 07:57:00.103564 6486 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:00.156544 6486 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:00.163568 6486 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 205678us May 06 07:57:00 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:00.173897 6486 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:57:00.191268 6486 fs_manager.cc:260] Metadata directory not provided I0506 07:57:00.191504 6486 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:00.191944 6486 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:57:00.192188 6486 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:00.215227 6486 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "96de26d68c2c4195bb9ef7225f964bd9" format_stamp: "Formatted at 2018-05-06 14:57:00 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:00.216281 6486 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "96de26d68c2c4195bb9ef7225f964bd9" format_stamp: "Formatted at 2018-05-06 14:57:00 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:00.262428 6486 fs_manager.cc:495] Time spent creating directory manager: real 0.046s user 0.015s sys 0.001s I0506 07:57:00.264264 6486 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:00.264921 6486 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:00.297605 6486 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s I0506 07:57:00.297797 6486 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/wal uuid: "96de26d68c2c4195bb9ef7225f964bd9" format_stamp: "Formatted at 2018-05-06 14:57:00 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:00.298112 6486 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:00.979176 6486 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:00.987269 6486 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:00.994930 6486 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:01.802830 6486 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:51760 I0506 07:57:01.803977 6486 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:57:01.814060 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 6486 I0506 07:57:01.828400 6486 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. May 06 07:57:01 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:57:01.885499 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:57:01.886467 6947 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:01.812444 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:45967 0506 07:57:01.825789 (+ 13345us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:01.825826 (+ 37us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:01.825899 (+ 73us) server_negotiation.cc:373] Connection header received 0506 07:57:01.844036 (+ 18137us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:01.844070 (+ 34us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:01.844314 (+ 244us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:01.844641 (+ 327us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:01.854119 (+ 9478us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:01.855031 (+ 912us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:01.857154 (+ 2123us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:01.859237 (+ 2083us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:01.859439 (+ 202us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:01.865458 (+ 6019us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:01.865536 (+ 78us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:01.865574 (+ 38us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:01.865665 (+ 91us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:01.873851 (+ 8186us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:01.876753 (+ 2902us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:01.876785 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:01.876808 (+ 23us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:01.877510 (+ 702us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:01.879243 (+ 1733us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:01.879272 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:01.879291 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:01.880002 (+ 711us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:01.880527 (+ 525us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:01.884104 (+ 3577us) server_negotiation.cc:264] Negotiation successful 0506 07:57:01.884491 (+ 387us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13041,"thread_start_us":889,"threads_started":1} I0506 07:57:01.888245 6948 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:01.812657 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51336 0506 07:57:01.825190 (+ 12533us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:01.825268 (+ 78us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:01.843500 (+ 18232us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:01.849520 (+ 6020us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:01.849554 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:01.850890 (+ 1336us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:01.853143 (+ 2253us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:01.853825 (+ 682us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:01.853882 (+ 57us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:01.855329 (+ 1447us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:01.855357 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:01.856913 (+ 1556us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:01.856954 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:01.859986 (+ 3032us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:01.860014 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:01.860470 (+ 456us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:01.860764 (+ 294us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:01.860786 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:01.864495 (+ 3709us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:01.874675 (+ 10180us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:01.874735 (+ 60us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:01.874762 (+ 27us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:01.876051 (+ 1289us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:01.878219 (+ 2168us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:01.878246 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:01.878267 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:01.878485 (+ 218us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:01.880991 (+ 2506us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:01.881023 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:01.881501 (+ 478us) client_negotiation.cc:695] Sending connection context 0506 07:57:01.882088 (+ 587us) client_negotiation.cc:225] Negotiation successful 0506 07:57:01.882458 (+ 370us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":12187,"thread_start_us":7340,"threads_started":1} I0506 07:57:01.892408 6943 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51336 I0506 07:57:01.892762 6943 heartbeater.cc:426] Registering TS with master... I0506 07:57:01.893373 6943 heartbeater.cc:475] Master 127.0.196.126:51336 requested a full tablet report, sending... I0506 07:57:01.896116 6387 ts_manager.cc:81] Registered new tserver with Master: 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760) I0506 07:57:01.899091 6387 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:45967 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:02.106457 6974 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:02.107110 6974 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:57:02.107587 6974 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:02.127390 6974 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:02.127663 6974 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:02.128013 6974 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:57:02.128679 6974 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:57:02.172017 6974 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:02.187712 6974 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:02.191064 6974 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 206678us May 06 07:57:02 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:02.198019 6974 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:57:02.217805 6974 fs_manager.cc:260] Metadata directory not provided I0506 07:57:02.218068 6974 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:02.218433 6974 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:57:02.218672 6974 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:02.234745 6974 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "36741b1b37e6414aa8fab04ae52b0afc" format_stamp: "Formatted at 2018-05-06 14:57:02 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:02.236064 6974 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "36741b1b37e6414aa8fab04ae52b0afc" format_stamp: "Formatted at 2018-05-06 14:57:02 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:02.269217 6974 fs_manager.cc:495] Time spent creating directory manager: real 0.033s user 0.013s sys 0.009s I0506 07:57:02.271100 6974 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:02.272528 6974 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:02.286545 6974 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.002s sys 0.006s I0506 07:57:02.286784 6974 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/wal uuid: "36741b1b37e6414aa8fab04ae52b0afc" format_stamp: "Formatted at 2018-05-06 14:57:02 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:02.287150 6974 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:02.652829 6974 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:02.679390 6974 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:02.696653 6974 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:02.902918 6943 heartbeater.cc:467] Master 127.0.196.126:51336 was elected leader, sending a full tablet report... I0506 07:57:03.146811 6974 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:42620 I0506 07:57:03.147569 6974 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:57:03.151715 6974 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:57:03.156682 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 6974 WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. May 06 07:57:03 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:03.203680 7330 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:03.153433 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51336 0506 07:57:03.157996 (+ 4563us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:03.158117 (+ 121us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:03.172684 (+ 14567us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:03.174476 (+ 1792us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:03.174511 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:03.175324 (+ 813us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:03.175597 (+ 273us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:03.175942 (+ 345us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:03.175985 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:03.178760 (+ 2775us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:03.178785 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:03.180296 (+ 1511us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:03.180345 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:03.182356 (+ 2011us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:03.182386 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:03.182884 (+ 498us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:03.183194 (+ 310us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:03.183215 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:03.186232 (+ 3017us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:03.193898 (+ 7666us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:03.193931 (+ 33us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:03.193956 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:03.195291 (+ 1335us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:03.197885 (+ 2594us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:03.197912 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:03.197932 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:03.198202 (+ 270us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:03.199767 (+ 1565us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:03.199796 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:03.200262 (+ 466us) client_negotiation.cc:695] Sending connection context 0506 07:57:03.200789 (+ 527us) client_negotiation.cc:225] Negotiation successful 0506 07:57:03.201186 (+ 397us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3890,"thread_start_us":3485,"threads_started":1} I0506 07:57:03.204583 7329 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:03.153810 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:58507 0506 07:57:03.172414 (+ 18604us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:03.172439 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:03.172479 (+ 40us) server_negotiation.cc:373] Connection header received 0506 07:57:03.173650 (+ 1171us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:03.173676 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:03.173814 (+ 138us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:03.173945 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:03.177974 (+ 4029us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:03.178549 (+ 575us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:03.180679 (+ 2130us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:03.182120 (+ 1441us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:03.182299 (+ 179us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:03.186819 (+ 4520us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:03.186890 (+ 71us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:03.186914 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:03.186984 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:03.193205 (+ 6221us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:03.196639 (+ 3434us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:03.196666 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:03.196683 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:03.197332 (+ 649us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:03.198728 (+ 1396us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:03.198755 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:03.198774 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:03.199242 (+ 468us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:03.199628 (+ 386us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:03.201084 (+ 1456us) server_negotiation.cc:264] Negotiation successful 0506 07:57:03.201439 (+ 355us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":18397,"thread_start_us":18635,"threads_started":1} I0506 07:57:03.207195 7323 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51336 I0506 07:57:03.207624 7323 heartbeater.cc:426] Registering TS with master... I0506 07:57:03.208245 7323 heartbeater.cc:475] Master 127.0.196.126:51336 requested a full tablet report, sending... I0506 07:57:03.210661 6387 ts_manager.cc:81] Registered new tserver with Master: 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66:42620) I0506 07:57:03.212616 6387 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:58507 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. I0506 07:57:03.217917 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:03.358781 7344 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:03.359869 7344 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:57:03.360316 7344 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:03.376344 7344 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:03.376564 7344 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:03.376885 7344 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:57:03.377435 7344 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:57:03.418601 7344 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51336 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:03.435833 7344 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:03.438489 7344 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 207178us May 06 07:57:03 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:03.442083 7344 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:57:03.451876 7344 fs_manager.cc:260] Metadata directory not provided I0506 07:57:03.452085 7344 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:03.452397 7344 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:57:03.454046 7344 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:03.468456 7344 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "56a9952cb7ae462b92bd50882285c768" format_stamp: "Formatted at 2018-05-06 14:57:03 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:03.469595 7344 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "56a9952cb7ae462b92bd50882285c768" format_stamp: "Formatted at 2018-05-06 14:57:03 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:03.484757 7344 fs_manager.cc:495] Time spent creating directory manager: real 0.015s user 0.011s sys 0.002s I0506 07:57:03.486277 7344 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:03.487000 7344 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:03.498980 7344 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0506 07:57:03.499143 7344 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/wal uuid: "56a9952cb7ae462b92bd50882285c768" format_stamp: "Formatted at 2018-05-06 14:57:03 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:03.499517 7344 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:03.913421 7344 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:03.920191 7344 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:03.922318 7344 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:04.217100 7323 heartbeater.cc:467] Master 127.0.196.126:51336 was elected leader, sending a full tablet report... I0506 07:57:04.684036 7344 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:52406 I0506 07:57:04.685622 7344 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:57:04.693472 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 7344 I0506 07:57:04.723157 7344 tablet_server_main.cc:87] Tablet server successfully started. May 06 07:57:04 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:04 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[6140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:04.763923 7619 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.695364 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:47989 0506 07:57:04.717930 (+ 22566us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.717954 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.717990 (+ 36us) server_negotiation.cc:373] Connection header received 0506 07:57:04.718098 (+ 108us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.718122 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.718251 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.719116 (+ 865us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:04.723300 (+ 4184us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.724027 (+ 727us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.726338 (+ 2311us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.727908 (+ 1570us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.728082 (+ 174us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.735789 (+ 7707us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:04.735818 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:04.735841 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:04.735918 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:04.748012 (+ 12094us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.749826 (+ 1814us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.749853 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.749870 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.751435 (+ 1565us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.752117 (+ 682us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.752145 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.752165 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.758978 (+ 6813us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:04.759367 (+ 389us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:04.761527 (+ 2160us) server_negotiation.cc:264] Negotiation successful 0506 07:57:04.762030 (+ 503us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":6247,"server-negotiator.queue_time_us":22375,"thread_start_us":22128,"threads_started":1} I0506 07:57:04.776291 7617 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.694162 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:38859 0506 07:57:04.704718 (+ 10556us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.704748 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.722609 (+ 17861us) server_negotiation.cc:373] Connection header received 0506 07:57:04.739117 (+ 16508us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.739144 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.739263 (+ 119us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.739440 (+ 177us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:04.742049 (+ 2609us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.742690 (+ 641us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.745431 (+ 2741us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.747055 (+ 1624us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.747226 (+ 171us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.752903 (+ 5677us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:04.752975 (+ 72us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:04.753001 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:04.753075 (+ 74us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:04.760059 (+ 6984us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.762949 (+ 2890us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.762977 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.762997 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.764273 (+ 1276us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.767473 (+ 3200us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.767499 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.767519 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.768087 (+ 568us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:04.768467 (+ 380us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:04.773402 (+ 4935us) server_negotiation.cc:264] Negotiation successful 0506 07:57:04.773751 (+ 349us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":10281,"thread_start_us":10005,"threads_started":1} I0506 07:57:04.777650 7616 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.692807 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51336 0506 07:57:04.722300 (+ 29493us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:04.722421 (+ 121us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:04.738790 (+ 16369us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:04.740058 (+ 1268us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:04.740095 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:04.740973 (+ 878us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:04.741277 (+ 304us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.741759 (+ 482us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.741818 (+ 59us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.742979 (+ 1161us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.743005 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.744677 (+ 1672us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.744723 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.747767 (+ 3044us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.747796 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.748255 (+ 459us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.748584 (+ 329us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:04.748631 (+ 47us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:04.751922 (+ 3291us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:04.760786 (+ 8864us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:04.760822 (+ 36us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:04.760850 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:04.762247 (+ 1397us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:04.766454 (+ 4207us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:04.766481 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:04.766501 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:04.766802 (+ 301us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:04.768929 (+ 2127us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:04.768960 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:04.769925 (+ 965us) client_negotiation.cc:695] Sending connection context 0506 07:57:04.770530 (+ 605us) client_negotiation.cc:225] Negotiation successful 0506 07:57:04.770950 (+ 420us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":28891,"thread_start_us":32040,"threads_started":1} I0506 07:57:04.780985 7615 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51336 I0506 07:57:04.781330 7615 heartbeater.cc:426] Registering TS with master... I0506 07:57:04.782053 7615 heartbeater.cc:475] Master 127.0.196.126:51336 requested a full tablet report, sending... I0506 07:57:04.784816 6387 ts_manager.cc:81] Registered new tserver with Master: 56a9952cb7ae462b92bd50882285c768 (127.0.196.67:52406) I0506 07:57:04.786816 6387 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:38859 I0506 07:57:04.788048 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0506 07:57:04.847349 7617 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.804484 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:47990 0506 07:57:04.804767 (+ 283us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.804791 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.808651 (+ 3860us) server_negotiation.cc:373] Connection header received 0506 07:57:04.808927 (+ 276us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.808947 (+ 20us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.809043 (+ 96us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.809159 (+ 116us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:04.811829 (+ 2670us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.812348 (+ 519us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.814748 (+ 2400us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.816128 (+ 1380us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.816291 (+ 163us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.832411 (+ 16120us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:04.832442 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:04.832465 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:04.832540 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:04.839197 (+ 6657us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.840997 (+ 1800us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.841025 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.841045 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.841743 (+ 698us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:04.842353 (+ 610us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:04.842381 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:04.842401 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:04.842932 (+ 531us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:04.843228 (+ 296us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:04.845100 (+ 1872us) server_negotiation.cc:264] Negotiation successful 0506 07:57:04.845557 (+ 457us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} I0506 07:57:04.856287 6387 catalog_manager.cc:1316] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.196.126:47990: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0506 07:57:04.888689 6387 catalog_manager.cc:1472] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0506 07:57:04.999613 7650 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.950750 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:51760 0506 07:57:04.966457 (+ 15707us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:04.966529 (+ 72us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:04.966750 (+ 221us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:04.978025 (+ 11275us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:04.978054 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:04.978111 (+ 57us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:57:04.978326 (+ 215us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.978548 (+ 222us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.978589 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.980994 (+ 2405us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.981030 (+ 36us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.989297 (+ 8267us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.989420 (+ 123us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.997060 (+ 7640us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.997089 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.997539 (+ 450us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.997698 (+ 159us) client_negotiation.cc:695] Sending connection context 0506 07:57:04.997824 (+ 126us) client_negotiation.cc:225] Negotiation successful 0506 07:57:04.998167 (+ 343us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15436,"thread_start_us":20918,"threads_started":1} I0506 07:57:05.003389 7652 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.951990 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:49448 0506 07:57:04.977057 (+ 25067us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.977088 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.977137 (+ 49us) server_negotiation.cc:373] Connection header received 0506 07:57:04.977289 (+ 152us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.977319 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.977500 (+ 181us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.977687 (+ 187us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:57:04.980013 (+ 2326us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.980752 (+ 739us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.989878 (+ 9126us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.994240 (+ 4362us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.994442 (+ 202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:04.994812 (+ 370us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.001121 (+ 6309us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.001454 (+ 333us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":24848,"thread_start_us":27714,"threads_started":1} I0506 07:57:05.007269 7656 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.951664 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:52406 0506 07:57:04.979413 (+ 27749us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:04.979495 (+ 82us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:04.979632 (+ 137us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:04.988957 (+ 9325us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:04.989048 (+ 91us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:04.989344 (+ 296us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:57:04.990004 (+ 660us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.990265 (+ 261us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.990310 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.993283 (+ 2973us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:04.993311 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.996753 (+ 3442us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.996796 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.004834 (+ 8038us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.004862 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.005306 (+ 444us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.005424 (+ 118us) client_negotiation.cc:695] Sending connection context 0506 07:57:05.005521 (+ 97us) client_negotiation.cc:225] Negotiation successful 0506 07:57:05.005856 (+ 335us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":27523,"thread_start_us":37278,"threads_started":1} I0506 07:57:05.010494 6785 tablet_service.cc:760] Processing CreateTablet for tablet 09a6703408c04e89b11faab3e63de86e (table=test-table [id=86a440fa7baf46f89a34dda4cf77ebe4]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:05.011495 6785 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 09a6703408c04e89b11faab3e63de86e. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:05.018270 7651 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.977192 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:42620 0506 07:57:04.990582 (+ 13390us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:04.990629 (+ 47us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:04.990789 (+ 160us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:04.991810 (+ 1021us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:04.991836 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:04.991898 (+ 62us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:57:04.992726 (+ 828us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:04.992983 (+ 257us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:04.993029 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.007753 (+ 14724us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.007784 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.010780 (+ 2996us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:05.010826 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.014508 (+ 3682us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.014533 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.015020 (+ 487us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.015139 (+ 119us) client_negotiation.cc:695] Sending connection context 0506 07:57:05.015575 (+ 436us) client_negotiation.cc:225] Negotiation successful 0506 07:57:05.016883 (+ 1308us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":12484,"thread_start_us":5175,"threads_started":1} I0506 07:57:05.020714 7654 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.972679 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:55968 0506 07:57:04.986196 (+ 13517us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.986224 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.991143 (+ 4919us) server_negotiation.cc:373] Connection header received 0506 07:57:04.991314 (+ 171us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.991342 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.991550 (+ 208us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.991746 (+ 196us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:57:05.004044 (+ 12298us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.007515 (+ 3471us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.011048 (+ 3533us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.014171 (+ 3123us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.018656 (+ 4485us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.019006 (+ 350us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.019136 (+ 130us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.019475 (+ 339us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13273,"thread_start_us":13426,"threads_started":1} I0506 07:57:05.021797 7653 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:04.952914 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:38580 0506 07:57:04.985263 (+ 32349us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:04.985292 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:04.985333 (+ 41us) server_negotiation.cc:373] Connection header received 0506 07:57:04.985473 (+ 140us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:04.985500 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:04.985725 (+ 225us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:04.985860 (+ 135us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:57:04.992177 (+ 6317us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:04.993032 (+ 855us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.000066 (+ 7034us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.003251 (+ 3185us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.003447 (+ 196us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.003891 (+ 444us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.008850 (+ 4959us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.010401 (+ 1551us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":32111,"thread_start_us":31762,"threads_started":1} I0506 07:57:05.024869 7519 tablet_service.cc:760] Processing CreateTablet for tablet 09a6703408c04e89b11faab3e63de86e (table=test-table [id=86a440fa7baf46f89a34dda4cf77ebe4]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:05.025876 7519 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 09a6703408c04e89b11faab3e63de86e. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:05.025632 7183 tablet_service.cc:760] Processing CreateTablet for tablet 09a6703408c04e89b11faab3e63de86e (table=test-table [id=86a440fa7baf46f89a34dda4cf77ebe4]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:05.026640 7183 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 09a6703408c04e89b11faab3e63de86e. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:05.043884 6785 ts_tablet_manager.cc:1173] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:05.064368 7183 ts_tablet_manager.cc:1173] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:05.065237 7662 ts_tablet_manager.cc:1028] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Bootstrapping tablet I0506 07:57:05.066848 7662 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Bootstrap starting. I0506 07:57:05.070109 7662 tablet_bootstrap.cc:587] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: No blocks or log segments found. Creating new log. I0506 07:57:05.075649 7665 ts_tablet_manager.cc:1028] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Bootstrapping tablet I0506 07:57:05.077253 7665 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Bootstrap starting. I0506 07:57:05.078933 7662 log.cc:520] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Log is configured to *not* fsync() on all Append() calls I0506 07:57:05.082259 7519 ts_tablet_manager.cc:1173] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:05.091356 7665 tablet_bootstrap.cc:587] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: No blocks or log segments found. Creating new log. I0506 07:57:05.093154 7665 log.cc:520] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Log is configured to *not* fsync() on all Append() calls I0506 07:57:05.094406 7669 ts_tablet_manager.cc:1028] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Bootstrapping tablet I0506 07:57:05.097189 7669 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Bootstrap starting. I0506 07:57:05.100914 7669 tablet_bootstrap.cc:587] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: No blocks or log segments found. Creating new log. I0506 07:57:05.114579 7662 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: No bootstrap required, opened a new log I0506 07:57:05.115021 7662 ts_tablet_manager.cc:1045] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s I0506 07:57:05.123551 7669 log.cc:520] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Log is configured to *not* fsync() on all Append() calls I0506 07:57:05.126186 7665 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: No bootstrap required, opened a new log I0506 07:57:05.126590 7665 ts_tablet_manager.cc:1045] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.012s I0506 07:57:05.142967 7662 raft_consensus.cc:303] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.144193 7662 raft_consensus.cc:329] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:05.144690 7662 raft_consensus.cc:604] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96de26d68c2c4195bb9ef7225f964bd9, State: Initialized, Role: FOLLOWER I0506 07:57:05.145686 7662 consensus_queue.cc:227] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.157821 7669 tablet_bootstrap.cc:436] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: No bootstrap required, opened a new log I0506 07:57:05.158243 7669 ts_tablet_manager.cc:1045] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Time spent bootstrapping tablet: real 0.064s user 0.006s sys 0.005s I0506 07:57:05.159936 7665 raft_consensus.cc:303] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.160796 7665 raft_consensus.cc:329] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:05.161006 7665 raft_consensus.cc:604] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36741b1b37e6414aa8fab04ae52b0afc, State: Initialized, Role: FOLLOWER I0506 07:57:05.162832 7662 ts_tablet_manager.cc:1073] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Time spent starting tablet: real 0.048s user 0.022s sys 0.001s I0506 07:57:05.166107 7665 consensus_queue.cc:227] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.184988 7665 ts_tablet_manager.cc:1073] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Time spent starting tablet: real 0.058s user 0.021s sys 0.003s I0506 07:57:05.193362 7675 raft_consensus.cc:435] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0506 07:57:05.193985 7675 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Snoozing failure detection for 1.596s (starting election) I0506 07:57:05.194308 7675 raft_consensus.cc:457] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.195847 7675 leader_election.cc:230] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 56a9952cb7ae462b92bd50882285c768 I0506 07:57:05.196564 7675 leader_election.cc:230] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 96de26d68c2c4195bb9ef7225f964bd9 I0506 07:57:05.202244 7669 raft_consensus.cc:303] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.203200 7669 raft_consensus.cc:329] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:05.203446 7669 raft_consensus.cc:604] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56a9952cb7ae462b92bd50882285c768, State: Initialized, Role: FOLLOWER I0506 07:57:05.204216 7669 consensus_queue.cc:227] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } I0506 07:57:05.218972 7615 heartbeater.cc:467] Master 127.0.196.126:51336 was elected leader, sending a full tablet report... I0506 07:57:05.221746 7669 ts_tablet_manager.cc:1073] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Time spent starting tablet: real 0.063s user 0.019s sys 0.005s I0506 07:57:05.223826 7653 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.197778 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:43943 0506 07:57:05.198125 (+ 347us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:05.198150 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:05.206748 (+ 8598us) server_negotiation.cc:373] Connection header received 0506 07:57:05.207508 (+ 760us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:05.207533 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:05.207659 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:05.207805 (+ 146us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:57:05.208971 (+ 1166us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.209528 (+ 557us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.213813 (+ 4285us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.216062 (+ 2249us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.216246 (+ 184us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.216529 (+ 283us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.222573 (+ 6044us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.222656 (+ 83us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":142} I0506 07:57:05.225893 7681 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.197114 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:52406 0506 07:57:05.206607 (+ 9493us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:05.206653 (+ 46us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:05.207286 (+ 633us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:05.208096 (+ 810us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:05.208122 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:05.208174 (+ 52us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:57:05.208420 (+ 246us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.208725 (+ 305us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:05.208767 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.209897 (+ 1130us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.209923 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.212500 (+ 2577us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:05.212543 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.216884 (+ 4341us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.216912 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.217368 (+ 456us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.217537 (+ 169us) client_negotiation.cc:695] Sending connection context 0506 07:57:05.224325 (+ 6788us) client_negotiation.cc:225] Negotiation successful 0506 07:57:05.224388 (+ 63us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9273,"threads_started":1} W0506 07:57:05.226068 7622 tablet.cc:1678] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:05.246076 7547 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "09a6703408c04e89b11faab3e63de86e" candidate_uuid: "36741b1b37e6414aa8fab04ae52b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a9952cb7ae462b92bd50882285c768" is_pre_election: true I0506 07:57:05.246644 7652 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.197760 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:41889 0506 07:57:05.198093 (+ 333us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:05.198118 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:05.226594 (+ 28476us) server_negotiation.cc:373] Connection header received 0506 07:57:05.226866 (+ 272us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:05.226894 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:05.227029 (+ 135us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:05.227140 (+ 111us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0506 07:57:05.237067 (+ 9927us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.237637 (+ 570us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.240549 (+ 2912us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.242904 (+ 2355us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.243089 (+ 185us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.243362 (+ 273us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.245494 (+ 2132us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.245573 (+ 79us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":130} I0506 07:57:05.246737 7547 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Snoozing failure detection for 1.975s (vote granted) I0506 07:57:05.247043 7547 raft_consensus.cc:2219] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36741b1b37e6414aa8fab04ae52b0afc in term 0. I0506 07:57:05.248644 7681 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.198401 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:51760 0506 07:57:05.226469 (+ 28068us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:05.226506 (+ 37us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:05.226640 (+ 134us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:05.227432 (+ 792us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:05.227459 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:05.227514 (+ 55us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0506 07:57:05.227691 (+ 177us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.236823 (+ 9132us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:05.236868 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.237923 (+ 1055us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.237951 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.240293 (+ 2342us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:05.240337 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.243615 (+ 3278us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.243644 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:05.244153 (+ 509us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.244675 (+ 522us) client_negotiation.cc:695] Sending connection context 0506 07:57:05.247182 (+ 2507us) client_negotiation.cc:225] Negotiation successful 0506 07:57:05.247251 (+ 69us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":27874,"mutex_wait_us":37,"thread_start_us":10276,"threads_started":1} I0506 07:57:05.255146 6827 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "09a6703408c04e89b11faab3e63de86e" candidate_uuid: "36741b1b37e6414aa8fab04ae52b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96de26d68c2c4195bb9ef7225f964bd9" is_pre_election: true I0506 07:57:05.255798 6827 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Snoozing failure detection for 1.650s (vote granted) I0506 07:57:05.256085 6827 raft_consensus.cc:2219] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36741b1b37e6414aa8fab04ae52b0afc in term 0. I0506 07:57:05.248736 7100 leader_election.cc:362] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 pre-election: Vote granted by peer 56a9952cb7ae462b92bd50882285c768 I0506 07:57:05.372712 7100 leader_election.cc:257] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0506 07:57:05.373193 7082 leader_election.cc:362] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 pre-election: Vote granted by peer 96de26d68c2c4195bb9ef7225f964bd9 I0506 07:57:05.374019 7675 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Snoozing failure detection for 2.076s (election complete) I0506 07:57:05.374388 7675 raft_consensus.cc:2511] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0506 07:57:05.374724 7675 raft_consensus.cc:435] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0506 07:57:05.375085 7675 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Snoozing failure detection for 2.080s (starting election) I0506 07:57:05.375423 7675 raft_consensus.cc:2756] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:05.380794 7675 raft_consensus.cc:457] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } W0506 07:57:05.373739 7100 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.196.67:52406, user_credentials={real_user=jenkins}} blocked reactor thread for 125107us W0506 07:57:05.382344 7082 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.196.65:51760, user_credentials={real_user=jenkins}} blocked reactor thread for 124909us I0506 07:57:05.382863 7675 leader_election.cc:230] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 election: Requesting vote from peer 56a9952cb7ae462b92bd50882285c768 I0506 07:57:05.384081 7675 leader_election.cc:230] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 election: Requesting vote from peer 96de26d68c2c4195bb9ef7225f964bd9 I0506 07:57:05.384812 7547 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "09a6703408c04e89b11faab3e63de86e" candidate_uuid: "36741b1b37e6414aa8fab04ae52b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56a9952cb7ae462b92bd50882285c768" I0506 07:57:05.385273 7547 raft_consensus.cc:2756] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:05.385558 7547 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Snoozing failure detection for 1.761s (vote granted) I0506 07:57:05.387522 6827 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "09a6703408c04e89b11faab3e63de86e" candidate_uuid: "36741b1b37e6414aa8fab04ae52b0afc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96de26d68c2c4195bb9ef7225f964bd9" I0506 07:57:05.388013 6827 raft_consensus.cc:2756] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:05.388305 6827 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Snoozing failure detection for 1.833s (vote granted) I0506 07:57:05.390242 7547 raft_consensus.cc:2219] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36741b1b37e6414aa8fab04ae52b0afc in term 1. I0506 07:57:05.392171 7100 leader_election.cc:362] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 election: Vote granted by peer 56a9952cb7ae462b92bd50882285c768 I0506 07:57:05.392379 7100 leader_election.cc:257] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:05.392905 7675 raft_consensus.cc:2704] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Snoozing failure detection for 1.661s (election complete) I0506 07:57:05.393151 7675 raft_consensus.cc:2511] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:05.395454 6827 raft_consensus.cc:2219] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36741b1b37e6414aa8fab04ae52b0afc in term 1. I0506 07:57:05.397627 7082 leader_election.cc:362] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [CANDIDATE]: Term 1 election: Vote granted by peer 96de26d68c2c4195bb9ef7225f964bd9 I0506 07:57:05.400674 7675 raft_consensus.cc:571] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 1 LEADER]: Becoming Leader. State: Replica: 36741b1b37e6414aa8fab04ae52b0afc, State: Running, Role: LEADER I0506 07:57:05.401302 7675 consensus_queue.cc:204] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } } W0506 07:57:05.403101 7325 tablet.cc:1678] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:05.417726 6387 catalog_manager.cc:3716] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc reported cstate change: term changed from 0 to 1, leader changed from to 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66). New cstate: current_term: 1 leader_uuid: "36741b1b37e6414aa8fab04ae52b0afc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36741b1b37e6414aa8fab04ae52b0afc" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 42620 } health_report { overall_health: HEALTHY } } } W0506 07:57:05.325852 6946 tablet.cc:1678] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:05.567023 7698 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.534615 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:55969 0506 07:57:05.549588 (+ 14973us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:05.549614 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:05.549651 (+ 37us) server_negotiation.cc:373] Connection header received 0506 07:57:05.549820 (+ 169us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:05.549844 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:05.550005 (+ 161us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:05.550132 (+ 127us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:05.558785 (+ 8653us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.559951 (+ 1166us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.562106 (+ 2155us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:05.563584 (+ 1478us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:05.563781 (+ 197us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:05.564749 (+ 968us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:05.565182 (+ 433us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:05.565321 (+ 139us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:05.565559 (+ 238us) server_negotiation.cc:264] Negotiation successful 0506 07:57:05.565768 (+ 209us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14764,"thread_start_us":14510,"threads_started":1} I0506 07:57:05.598206 6827 raft_consensus.cc:1057] T 09a6703408c04e89b11faab3e63de86e P 96de26d68c2c4195bb9ef7225f964bd9 [term 1 FOLLOWER]: Refusing update from remote peer 36741b1b37e6414aa8fab04ae52b0afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:05.600111 7675 consensus_queue.cc:963] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96de26d68c2c4195bb9ef7225f964bd9" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 51760 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:05.606212 7547 raft_consensus.cc:1057] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 1 FOLLOWER]: Refusing update from remote peer 36741b1b37e6414aa8fab04ae52b0afc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:05.608130 7675 consensus_queue.cc:963] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56a9952cb7ae462b92bd50882285c768" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 52406 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:05.697005 6385 catalog_manager.cc:1713] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.196.126:47990: table { table_name: "test-table" } I0506 07:57:05.703419 6385 catalog_manager.cc:3927] T 00000000000000000000000000000000 P 3b63a104e88347dbb42b826d60c0c1e5: Sending DeleteTablet for 3 replicas of tablet 09a6703408c04e89b11faab3e63de86e I0506 07:57:05.703794 6385 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 56a9952cb7ae462b92bd50882285c768 (127.0.196.67:52406) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.704257 6385 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.704674 6385 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66:42620) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.705062 7519 tablet_service.cc:801] Processing DeleteTablet for tablet 09a6703408c04e89b11faab3e63de86e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:57:05 PDT) from {username='kudu', principal='kudu/127.0.196.126@KRBTEST.COM'} at 127.0.196.67:38580 I0506 07:57:05.706928 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 6486 I0506 07:57:05.709156 7183 tablet_service.cc:801] Processing DeleteTablet for tablet 09a6703408c04e89b11faab3e63de86e with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:57:05 PDT) from {username='kudu', principal='kudu/127.0.196.126@KRBTEST.COM'} at 127.0.196.66:55968 I0506 07:57:05.732568 7720 tablet_replica.cc:261] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: stopping tablet replica I0506 07:57:05.733099 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op CompactRowSetsOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.733376 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op MinorDeltaCompactionOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.733592 7719 tablet_replica.cc:261] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: stopping tablet replica I0506 07:57:05.733650 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op MajorDeltaCompactionOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.733839 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op UndoDeltaBlockGCOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.733947 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op CompactRowSetsOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734077 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op FlushMRSOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734256 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op MinorDeltaCompactionOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734266 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op FlushDeltaMemStoresOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734437 7720 maintenance_manager.cc:237] P 36741b1b37e6414aa8fab04ae52b0afc: Unregistered op LogGCOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734473 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op MajorDeltaCompactionOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734678 7720 raft_consensus.cc:2016] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 1 LEADER]: Raft consensus shutting down. I0506 07:57:05.734745 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op UndoDeltaBlockGCOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.734912 7720 consensus_peers.cc:456] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc -> Peer 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760): Closing peer: 96de26d68c2c4195bb9ef7225f964bd9 I0506 07:57:05.735021 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op FlushMRSOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.735159 7720 consensus_peers.cc:456] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc -> Peer 56a9952cb7ae462b92bd50882285c768 (127.0.196.67:52406): Closing peer: 56a9952cb7ae462b92bd50882285c768 I0506 07:57:05.735285 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op FlushDeltaMemStoresOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.735528 7719 maintenance_manager.cc:237] P 56a9952cb7ae462b92bd50882285c768: Unregistered op LogGCOp(09a6703408c04e89b11faab3e63de86e) I0506 07:57:05.735755 7720 raft_consensus.cc:2043] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:57:05.735900 7719 raft_consensus.cc:2016] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 1 FOLLOWER]: Raft consensus shutting down. I0506 07:57:05.736825 7719 raft_consensus.cc:2043] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768 [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:57:05.740290 7719 ts_tablet_manager.cc:1369] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:57:05.760285 7720 ts_tablet_manager.cc:1369] T 09a6703408c04e89b11faab3e63de86e P 36741b1b37e6414aa8fab04ae52b0afc: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:57:05.772976 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 6974 W0506 07:57:05.780112 6271 connection.cc:512] server connection from 127.0.196.65:45967 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:45967): Connection reset by peer (error 104) W0506 07:57:05.780959 6271 catalog_manager.cc:2785] TS 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760): Delete Tablet RPC failed for tablet 09a6703408c04e89b11faab3e63de86e: Network error: recv got EOF from 127.0.196.65:51760 (error 108) I0506 07:57:05.781195 6271 catalog_manager.cc:2821] Scheduling retry of 09a6703408c04e89b11faab3e63de86e Delete Tablet RPC for TS=96de26d68c2c4195bb9ef7225f964bd9 with a delay of 32 ms (attempt = 1) I0506 07:57:05.791313 7719 ts_tablet_manager.cc:1382] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:57:05.791636 7719 log.cc:981] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1525618592290500-785/minicluster-data/ts-2/wal/wals/09a6703408c04e89b11faab3e63de86e I0506 07:57:05.792307 7719 ts_tablet_manager.cc:1402] T 09a6703408c04e89b11faab3e63de86e P 56a9952cb7ae462b92bd50882285c768: Deleting consensus metadata I0506 07:57:05.794904 6333 catalog_manager.cc:3021] TS 56a9952cb7ae462b92bd50882285c768 (127.0.196.67:52406): tablet 09a6703408c04e89b11faab3e63de86e (table test-table [id=86a440fa7baf46f89a34dda4cf77ebe4]) successfully deleted I0506 07:57:05.814090 6271 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.817600 7731 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.814677 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:51760 0506 07:57:05.815645 (+ 968us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:05.817078 (+ 1433us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.65:51760: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":430,"threads_started":1} W0506 07:57:05.819222 6271 catalog_manager.cc:2785] TS 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760): Delete Tablet RPC failed for tablet 09a6703408c04e89b11faab3e63de86e: Network error: Client connection negotiation failed: client connection to 127.0.196.65:51760: connect: Connection refused (error 111) I0506 07:57:05.819454 6271 catalog_manager.cc:2821] Scheduling retry of 09a6703408c04e89b11faab3e63de86e Delete Tablet RPC for TS=96de26d68c2c4195bb9ef7225f964bd9 with a delay of 79 ms (attempt = 2) W0506 07:57:05.833124 6271 catalog_manager.cc:2785] TS 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66:42620): Delete Tablet RPC failed for tablet 09a6703408c04e89b11faab3e63de86e: Network error: recv got EOF from 127.0.196.66:42620 (error 108) I0506 07:57:05.833370 6271 catalog_manager.cc:2821] Scheduling retry of 09a6703408c04e89b11faab3e63de86e Delete Tablet RPC for TS=36741b1b37e6414aa8fab04ae52b0afc with a delay of 64 ms (attempt = 1) W0506 07:57:05.834055 6267 connection.cc:512] server connection from 127.0.196.66:58507 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:58507): Connection reset by peer (error 104) W0506 07:57:05.834959 7389 connection.cc:512] server connection from 127.0.196.66:43943 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:43943): Connection reset by peer (error 104) I0506 07:57:05.836165 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 7344 I0506 07:57:05.898286 6271 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66:42620) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.899657 7731 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.898779 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:42620 0506 07:57:05.899061 (+ 282us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:05.899216 (+ 155us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.66:42620: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":91} W0506 07:57:05.900127 6271 catalog_manager.cc:2785] TS 36741b1b37e6414aa8fab04ae52b0afc (127.0.196.66:42620): Delete Tablet RPC failed for tablet 09a6703408c04e89b11faab3e63de86e: Network error: Client connection negotiation failed: client connection to 127.0.196.66:42620: connect: Connection refused (error 111) I0506 07:57:05.900388 6271 catalog_manager.cc:2821] Scheduling retry of 09a6703408c04e89b11faab3e63de86e Delete Tablet RPC for TS=36741b1b37e6414aa8fab04ae52b0afc with a delay of 79 ms (attempt = 2) I0506 07:57:05.900807 6271 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 09a6703408c04e89b11faab3e63de86e on 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760) (Table deleted at 2018-05-06 07:57:05 PDT) I0506 07:57:05.904372 7731 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:05.901322 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:51760 0506 07:57:05.903779 (+ 2457us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:05.903950 (+ 171us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.65:51760: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":2244} W0506 07:57:05.904811 6271 catalog_manager.cc:2785] TS 96de26d68c2c4195bb9ef7225f964bd9 (127.0.196.65:51760): Delete Tablet RPC failed for tablet 09a6703408c04e89b11faab3e63de86e: Network error: Client connection negotiation failed: client connection to 127.0.196.65:51760: connect: Connection refused (error 111) I0506 07:57:05.905043 6271 catalog_manager.cc:2821] Scheduling retry of 09a6703408c04e89b11faab3e63de86e Delete Tablet RPC for TS=96de26d68c2c4195bb9ef7225f964bd9 with a delay of 86 ms (attempt = 3) W0506 07:57:05.906944 6265 connection.cc:512] server connection from 127.0.196.67:38859 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:38859): Connection reset by peer (error 104) I0506 07:57:05.907763 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 6186 [ OK ] SecurityITest.TestDisableWebUI (8528 ms) [ RUN ] SecurityITest.TestDisableAuthenticationEncryption I0506 07:57:06.062187 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:52549 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:06.488304 7757 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:06.488945 7757 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:06.540256 7757 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:06.540455 7757 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:06.540638 7757 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:06.540801 7757 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:06.540987 7757 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true I0506 07:57:06.639307 7757 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:52549 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:06.659097 7757 master_main.cc:73] Initializing master server... I0506 07:57:06.663837 7757 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 208678us I0506 07:57:06.676506 7757 fs_manager.cc:260] Metadata directory not provided I0506 07:57:06.676795 7757 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:06.679569 7757 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:06.679858 7757 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:06.698711 7757 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "a484ae26c7c442229ecfb87426210e16" format_stamp: "Formatted at 2018-05-06 14:57:06 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:06.699806 7757 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "a484ae26c7c442229ecfb87426210e16" format_stamp: "Formatted at 2018-05-06 14:57:06 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:06.735152 7757 fs_manager.cc:495] Time spent creating directory manager: real 0.035s user 0.012s sys 0.011s I0506 07:57:06.737233 7757 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:06.738761 7757 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:06.767524 7757 fs_manager.cc:417] Time spent opening block manager: real 0.015s user 0.001s sys 0.004s I0506 07:57:06.767808 7757 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/wal uuid: "a484ae26c7c442229ecfb87426210e16" format_stamp: "Formatted at 2018-05-06 14:57:06 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:06.768236 7757 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:07.025018 7757 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:07.030619 7757 master_main.cc:76] Starting Master server... I0506 07:57:07.434959 7757 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:52549 I0506 07:57:07.436221 7757 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:07.436332 7757 webserver.cc:184] Document root disabled I0506 07:57:07.439682 7757 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:36122/ I0506 07:57:07.440624 7757 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:07.441848 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 7757 I0506 07:57:07.443120 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0506 07:57:07.448901 7917 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:07.472812 7917 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Bootstrap starting. I0506 07:57:07.476672 7917 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: No blocks or log segments found. Creating new log. I0506 07:57:07.478909 7917 log.cc:520] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Log is configured to *not* fsync() on all Append() calls I0506 07:57:07.487582 7917 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: No bootstrap required, opened a new log I0506 07:57:07.506963 7917 raft_consensus.cc:303] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } I0506 07:57:07.507462 7917 raft_consensus.cc:329] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:07.507724 7917 raft_consensus.cc:604] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a484ae26c7c442229ecfb87426210e16, State: Initialized, Role: FOLLOWER I0506 07:57:07.508381 7917 consensus_queue.cc:227] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } I0506 07:57:07.508934 7917 raft_consensus.cc:343] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:07.509202 7917 raft_consensus.cc:435] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:07.509883 7917 raft_consensus.cc:2704] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Snoozing failure detection for 1.553s (starting election) I0506 07:57:07.510130 7917 raft_consensus.cc:2756] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:07.514088 7917 raft_consensus.cc:457] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } I0506 07:57:07.515319 7917 leader_election.cc:257] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:07.522351 7929 raft_consensus.cc:2704] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Snoozing failure detection for 1.768s (election complete) I0506 07:57:07.524740 7929 raft_consensus.cc:2511] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:07.537991 7929 raft_consensus.cc:571] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [term 1 LEADER]: Becoming Leader. State: Replica: a484ae26c7c442229ecfb87426210e16, State: Running, Role: LEADER I0506 07:57:07.539412 7929 consensus_queue.cc:204] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } I0506 07:57:07.551962 7932 sys_catalog.cc:335] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a484ae26c7c442229ecfb87426210e16. Latest consensus state: current_term: 1 leader_uuid: "a484ae26c7c442229ecfb87426210e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } } I0506 07:57:07.552515 7932 sys_catalog.cc:338] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: This master's current role is: LEADER I0506 07:57:07.552641 7931 sys_catalog.cc:335] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a484ae26c7c442229ecfb87426210e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } } I0506 07:57:07.651499 7931 sys_catalog.cc:338] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: This master's current role is: LEADER I0506 07:57:07.552791 7917 sys_catalog.cc:335] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a484ae26c7c442229ecfb87426210e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a484ae26c7c442229ecfb87426210e16" member_type: VOTER } } I0506 07:57:07.656563 7917 sys_catalog.cc:338] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: This master's current role is: LEADER I0506 07:57:07.658272 7917 sys_catalog.cc:422] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:07.668828 7757 master_main.cc:79] Master server successfully started. I0506 07:57:07.668056 7937 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:07.669236 7934 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:07.675909 7934 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:07.676417 7919 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:07.677165 7919 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:07.689642 7919 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:07.689843 7919 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:07.690405 7919 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:57:07.729225 7919 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:07.739643 7934 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:07.740221 7934 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:07.758937 7919 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:07.761749 7919 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 209178us I0506 07:57:07.765815 7919 fs_manager.cc:260] Metadata directory not provided I0506 07:57:07.766049 7919 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:07.767223 7919 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:57:07.767473 7919 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:07.776768 7934 catalog_manager.cc:3992] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Generated new TSK 0 I0506 07:57:07.785992 7919 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "2c3fe82a8554492ab39c0826fb959f6f" format_stamp: "Formatted at 2018-05-06 14:57:07 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:07.787111 7919 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "2c3fe82a8554492ab39c0826fb959f6f" format_stamp: "Formatted at 2018-05-06 14:57:07 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:07.816180 7919 fs_manager.cc:495] Time spent creating directory manager: real 0.029s user 0.014s sys 0.010s I0506 07:57:07.818096 7919 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:07.819245 7919 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:07.837395 7919 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.000s sys 0.005s I0506 07:57:07.837605 7919 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/wal uuid: "2c3fe82a8554492ab39c0826fb959f6f" format_stamp: "Formatted at 2018-05-06 14:57:07 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:07.837991 7919 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:08.110065 7919 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:08.119428 7919 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:08.126018 7919 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:08.766363 7919 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:39558 I0506 07:57:08.767805 7919 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:57:08.768013 7919 webserver.cc:184] Document root disabled I0506 07:57:08.774013 7919 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:44007/ I0506 07:57:08.777057 7919 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:57:08.779631 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 7919 I0506 07:57:08.780995 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0506 07:57:08.801564 7919 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:57:08.819072 8281 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:08.783248 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:52549 0506 07:57:08.794276 (+ 11028us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:08.794353 (+ 77us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:08.814972 (+ 20619us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:08.816028 (+ 1056us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:08.816065 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:08.816169 (+ 104us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:08.816301 (+ 132us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:08.816324 (+ 23us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:08.816374 (+ 50us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:08.816541 (+ 167us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:08.817109 (+ 568us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:08.817141 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:08.817168 (+ 27us) client_negotiation.cc:695] Sending connection context 0506 07:57:08.817328 (+ 160us) client_negotiation.cc:225] Negotiation successful 0506 07:57:08.817515 (+ 187us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10619,"thread_start_us":11569,"threads_started":1} I0506 07:57:08.819087 8282 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:08.785098 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:55765 0506 07:57:08.799178 (+ 14080us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:08.799220 (+ 42us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:08.799521 (+ 301us) server_negotiation.cc:373] Connection header received 0506 07:57:08.815438 (+ 15917us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:08.815464 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:08.815610 (+ 146us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:08.815887 (+ 277us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:08.816799 (+ 912us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:08.816824 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:08.816842 (+ 18us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:08.816873 (+ 31us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:08.816943 (+ 70us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:08.816983 (+ 40us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:08.817094 (+ 111us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:08.817450 (+ 356us) server_negotiation.cc:264] Negotiation successful 0506 07:57:08.817620 (+ 170us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13722,"thread_start_us":7854,"threads_started":1} I0506 07:57:08.821930 8280 heartbeater.cc:346] Connected to a master server at 127.0.196.126:52549 I0506 07:57:08.822226 8280 heartbeater.cc:426] Registering TS with master... I0506 07:57:08.822724 8280 heartbeater.cc:475] Master 127.0.196.126:52549 requested a full tablet report, sending... I0506 07:57:08.824662 7834 ts_manager.cc:81] Registered new tserver with Master: 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65:39558) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:08.939927 8283 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:08.940369 8283 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:08.949638 8283 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:08.949785 8283 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:08.950139 8283 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:57:08.966100 8283 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:08.975852 8283 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:08.978040 8283 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 209678us I0506 07:57:08.978509 8283 fs_manager.cc:260] Metadata directory not provided I0506 07:57:08.978667 8283 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:08.978997 8283 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:57:08.979216 8283 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:08.987993 8283 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "069f235f3a0a42e4a371cb34d113e46d" format_stamp: "Formatted at 2018-05-06 14:57:08 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:08.988718 8283 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "069f235f3a0a42e4a371cb34d113e46d" format_stamp: "Formatted at 2018-05-06 14:57:08 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:09.001117 8283 fs_manager.cc:495] Time spent creating directory manager: real 0.012s user 0.008s sys 0.004s I0506 07:57:09.002563 8283 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:09.003178 8283 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:09.010993 8283 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0506 07:57:09.011117 8283 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/wal uuid: "069f235f3a0a42e4a371cb34d113e46d" format_stamp: "Formatted at 2018-05-06 14:57:08 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:09.011342 8283 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:09.213207 8283 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:09.216843 8283 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:09.218205 8283 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:09.426268 8283 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:32803 I0506 07:57:09.427501 8283 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:57:09.427628 8283 webserver.cc:184] Document root disabled I0506 07:57:09.429309 8283 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:40415/ I0506 07:57:09.430114 8283 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:57:09.430704 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 8283 I0506 07:57:09.431970 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0506 07:57:09.451140 8283 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:57:09.474982 8412 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:09.449421 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:47322 0506 07:57:09.454530 (+ 5109us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:09.454558 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:09.456001 (+ 1443us) server_negotiation.cc:373] Connection header received 0506 07:57:09.471215 (+ 15214us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:09.471243 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:09.471345 (+ 102us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:09.471522 (+ 177us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:09.472890 (+ 1368us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:09.472917 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:09.472939 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:09.472983 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:09.473041 (+ 58us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:09.473101 (+ 60us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:09.473194 (+ 93us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:09.473721 (+ 527us) server_negotiation.cc:264] Negotiation successful 0506 07:57:09.473812 (+ 91us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4844,"thread_start_us":4330,"threads_started":1} I0506 07:57:09.478497 8411 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:09.447349 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:52549 0506 07:57:09.455795 (+ 8446us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:09.455874 (+ 79us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:09.470894 (+ 15020us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:09.471982 (+ 1088us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:09.472017 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:09.472128 (+ 111us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:09.472273 (+ 145us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:09.472296 (+ 23us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:09.472351 (+ 55us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:09.472520 (+ 169us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:09.473347 (+ 827us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:09.473378 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:09.473403 (+ 25us) client_negotiation.cc:695] Sending connection context 0506 07:57:09.476155 (+ 2752us) client_negotiation.cc:225] Negotiation successful 0506 07:57:09.476355 (+ 200us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7843,"thread_start_us":8465,"threads_started":1} I0506 07:57:09.480244 8408 heartbeater.cc:346] Connected to a master server at 127.0.196.126:52549 I0506 07:57:09.480571 8408 heartbeater.cc:426] Registering TS with master... I0506 07:57:09.481117 8408 heartbeater.cc:475] Master 127.0.196.126:52549 requested a full tablet report, sending... I0506 07:57:09.482718 7834 ts_manager.cc:81] Registered new tserver with Master: 069f235f3a0a42e4a371cb34d113e46d (127.0.196.66:32803) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:09.579650 8409 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:09.580117 8409 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:09.593852 8409 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:09.594063 8409 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:09.594619 8409 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:57:09.627148 8409 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:52549 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:09.636904 8409 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:09.639222 8409 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 210178us I0506 07:57:09.639719 8409 fs_manager.cc:260] Metadata directory not provided I0506 07:57:09.639868 8409 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:09.640100 8409 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:57:09.640254 8409 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:09.649503 8409 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "ca9a4e4b50a948dca456e1e9a3669b19" format_stamp: "Formatted at 2018-05-06 14:57:09 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:09.650470 8409 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "ca9a4e4b50a948dca456e1e9a3669b19" format_stamp: "Formatted at 2018-05-06 14:57:09 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:09.664659 8409 fs_manager.cc:495] Time spent creating directory manager: real 0.014s user 0.008s sys 0.005s I0506 07:57:09.665869 8409 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:09.666303 8409 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:09.675205 8409 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0506 07:57:09.675330 8409 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/wal uuid: "ca9a4e4b50a948dca456e1e9a3669b19" format_stamp: "Formatted at 2018-05-06 14:57:09 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:09.675554 8409 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:09.827049 8280 heartbeater.cc:467] Master 127.0.196.126:52549 was elected leader, sending a full tablet report... I0506 07:57:09.921681 8409 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:09.926723 8409 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:09.935613 8409 tablet_server_main.cc:84] Starting tablet server... I0506 07:57:10.272614 8409 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:43040 I0506 07:57:10.274526 8409 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:57:10.274674 8409 webserver.cc:184] Document root disabled I0506 07:57:10.277312 8409 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:60155/ I0506 07:57:10.278107 8409 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:57:10.284329 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 8409 I0506 07:57:10.285543 8409 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:57:10.297523 8578 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.287957 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34122 0506 07:57:10.295151 (+ 7194us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.295169 (+ 18us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.295194 (+ 25us) server_negotiation.cc:373] Connection header received 0506 07:57:10.295268 (+ 74us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.295282 (+ 14us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.295351 (+ 69us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.295469 (+ 118us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.296042 (+ 573us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.296061 (+ 19us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.296073 (+ 12us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.296102 (+ 29us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.296143 (+ 41us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.296175 (+ 32us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.296264 (+ 89us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.296535 (+ 271us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.296645 (+ 110us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7026,"thread_start_us":6802,"threads_started":1} I0506 07:57:10.312213 8577 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.286473 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:56385 0506 07:57:10.292240 (+ 5767us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.292268 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.295141 (+ 2873us) server_negotiation.cc:373] Connection header received 0506 07:57:10.308690 (+ 13549us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.308716 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.308816 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.308954 (+ 138us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.310318 (+ 1364us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.310345 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.310368 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.310412 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.310471 (+ 59us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.310523 (+ 52us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.310658 (+ 135us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.310935 (+ 277us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.311022 (+ 87us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5479,"thread_start_us":5186,"threads_started":1} I0506 07:57:10.312638 8576 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.286037 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:52549 0506 07:57:10.294386 (+ 8349us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.294506 (+ 120us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.308386 (+ 13880us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.309224 (+ 838us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.309256 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.309367 (+ 111us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.309523 (+ 156us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.309545 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.309637 (+ 92us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.310074 (+ 437us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.310640 (+ 566us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.310659 (+ 19us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.310681 (+ 22us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.310826 (+ 145us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.310959 (+ 133us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7548,"thread_start_us":6464,"threads_started":1} I0506 07:57:10.314988 8573 heartbeater.cc:346] Connected to a master server at 127.0.196.126:52549 I0506 07:57:10.315274 8573 heartbeater.cc:426] Registering TS with master... I0506 07:57:10.315867 8573 heartbeater.cc:475] Master 127.0.196.126:52549 requested a full tablet report, sending... I0506 07:57:10.317392 7834 ts_manager.cc:81] Registered new tserver with Master: ca9a4e4b50a948dca456e1e9a3669b19 (127.0.196.67:43040) I0506 07:57:10.319455 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0506 07:57:10.334313 8577 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.328858 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:34123 0506 07:57:10.329137 (+ 279us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.329160 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.331216 (+ 2056us) server_negotiation.cc:373] Connection header received 0506 07:57:10.331423 (+ 207us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.331450 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.331555 (+ 105us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.331702 (+ 147us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.332174 (+ 472us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.332204 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.332229 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.332273 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.332333 (+ 60us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.332390 (+ 57us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.332504 (+ 114us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.332734 (+ 230us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.332821 (+ 87us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":102} I0506 07:57:10.339601 7832 catalog_manager.cc:1316] Servicing CreateTable request from {username='jenkins'} at 127.0.196.126:34123: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0506 07:57:10.350749 7832 catalog_manager.cc:1472] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0506 07:57:10.389019 8590 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.376471 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:43282 0506 07:57:10.378745 (+ 2274us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.378775 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.383491 (+ 4716us) server_negotiation.cc:373] Connection header received 0506 07:57:10.383729 (+ 238us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.383760 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.383935 (+ 175us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.384089 (+ 154us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.385942 (+ 1853us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.385979 (+ 37us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.386012 (+ 33us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.386057 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.386168 (+ 111us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.386238 (+ 70us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.386369 (+ 131us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.387463 (+ 1094us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.387586 (+ 123us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":1048,"threads_started":1} I0506 07:57:10.391432 8589 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.377855 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:32803 0506 07:57:10.383321 (+ 5466us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.383398 (+ 77us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.383515 (+ 117us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.384762 (+ 1247us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.384788 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.384849 (+ 61us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.384933 (+ 84us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.384952 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.385020 (+ 68us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.385131 (+ 111us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.386600 (+ 1469us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.386651 (+ 51us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.386679 (+ 28us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.389546 (+ 2867us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.389699 (+ 153us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5175,"thread_start_us":2178,"threads_started":1} I0506 07:57:10.392627 8588 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.373619 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:56118 0506 07:57:10.375013 (+ 1394us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.375048 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.381197 (+ 6149us) server_negotiation.cc:373] Connection header received 0506 07:57:10.381630 (+ 433us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.381659 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.381851 (+ 192us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.382010 (+ 159us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.386853 (+ 4843us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.386887 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.386914 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.386958 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.387047 (+ 89us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.387104 (+ 57us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.387463 (+ 359us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.389976 (+ 2513us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.390086 (+ 110us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":1956,"threads_started":1} I0506 07:57:10.393208 8591 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.373210 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:39558 0506 07:57:10.380722 (+ 7512us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.381035 (+ 313us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.381383 (+ 348us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.382885 (+ 1502us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.382925 (+ 40us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.383020 (+ 95us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.383117 (+ 97us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.383146 (+ 29us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.383226 (+ 80us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.383359 (+ 133us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.387616 (+ 4257us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.387644 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.387666 (+ 22us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.387759 (+ 93us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.388277 (+ 518us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7128,"thread_start_us":11628,"threads_started":1} I0506 07:57:10.395004 8345 tablet_service.cc:760] Processing CreateTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 (table=test-table [id=de69b0a907d54bd4b4f184cb25929755]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:10.396116 8345 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1717e9a2c794b599c375e90a9ded6a6. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:10.396751 8133 tablet_service.cc:760] Processing CreateTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 (table=test-table [id=de69b0a907d54bd4b4f184cb25929755]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:10.397670 8133 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1717e9a2c794b599c375e90a9ded6a6. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:10.397770 8593 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.381119 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:38281 0506 07:57:10.393005 (+ 11886us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.393040 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.393087 (+ 47us) server_negotiation.cc:373] Connection header received 0506 07:57:10.393212 (+ 125us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.393243 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.393413 (+ 170us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.394077 (+ 664us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.395257 (+ 1180us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.395294 (+ 37us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.395321 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.395367 (+ 46us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.395471 (+ 104us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.395537 (+ 66us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.395704 (+ 167us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.396203 (+ 499us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.396319 (+ 116us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11633,"thread_start_us":12522,"threads_started":1} I0506 07:57:10.398576 8592 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.380881 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:43040 0506 07:57:10.387109 (+ 6228us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.387183 (+ 74us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.387295 (+ 112us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.394144 (+ 6849us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.394168 (+ 24us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.394238 (+ 70us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.394343 (+ 105us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.394362 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.394397 (+ 35us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.394493 (+ 96us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.395723 (+ 1230us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.395753 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.395777 (+ 24us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.395879 (+ 102us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.396430 (+ 551us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5944,"spinlock_wait_cycles":8576,"thread_start_us":5893,"threads_started":1} I0506 07:57:10.402247 8496 tablet_service.cc:760] Processing CreateTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 (table=test-table [id=de69b0a907d54bd4b4f184cb25929755]), partition=RANGE (key) PARTITION UNBOUNDED I0506 07:57:10.403295 8496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b1717e9a2c794b599c375e90a9ded6a6. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:10.414980 8133 ts_tablet_manager.cc:1173] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:10.418823 8594 ts_tablet_manager.cc:1028] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Bootstrapping tablet I0506 07:57:10.419215 8594 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Bootstrap starting. I0506 07:57:10.421403 8594 tablet_bootstrap.cc:587] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: No blocks or log segments found. Creating new log. I0506 07:57:10.423379 8345 ts_tablet_manager.cc:1173] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:10.424471 8594 log.cc:520] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Log is configured to *not* fsync() on all Append() calls I0506 07:57:10.428912 8595 ts_tablet_manager.cc:1028] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Bootstrapping tablet I0506 07:57:10.429514 8595 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Bootstrap starting. I0506 07:57:10.433179 8496 ts_tablet_manager.cc:1173] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Registered tablet (data state: TABLET_DATA_READY) I0506 07:57:10.433444 8595 tablet_bootstrap.cc:587] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: No blocks or log segments found. Creating new log. I0506 07:57:10.434747 8594 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: No bootstrap required, opened a new log I0506 07:57:10.435734 8595 log.cc:520] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Log is configured to *not* fsync() on all Append() calls I0506 07:57:10.437300 8594 ts_tablet_manager.cc:1045] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.005s I0506 07:57:10.440253 8597 ts_tablet_manager.cc:1028] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Bootstrapping tablet I0506 07:57:10.440855 8597 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Bootstrap starting. I0506 07:57:10.445097 8597 tablet_bootstrap.cc:587] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: No blocks or log segments found. Creating new log. I0506 07:57:10.446753 8597 log.cc:520] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Log is configured to *not* fsync() on all Append() calls I0506 07:57:10.450287 8595 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: No bootstrap required, opened a new log I0506 07:57:10.450739 8595 ts_tablet_manager.cc:1045] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.011s I0506 07:57:10.459045 8594 raft_consensus.cc:303] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.461599 8594 raft_consensus.cc:329] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:10.461880 8594 raft_consensus.cc:604] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3fe82a8554492ab39c0826fb959f6f, State: Initialized, Role: FOLLOWER I0506 07:57:10.462572 8597 tablet_bootstrap.cc:436] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: No bootstrap required, opened a new log I0506 07:57:10.463030 8597 ts_tablet_manager.cc:1045] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s I0506 07:57:10.462676 8594 consensus_queue.cc:227] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.470038 8595 raft_consensus.cc:303] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.471489 8595 raft_consensus.cc:329] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:10.471760 8595 raft_consensus.cc:604] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069f235f3a0a42e4a371cb34d113e46d, State: Initialized, Role: FOLLOWER I0506 07:57:10.472498 8595 consensus_queue.cc:227] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.478271 8594 ts_tablet_manager.cc:1073] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Time spent starting tablet: real 0.041s user 0.017s sys 0.008s I0506 07:57:10.483027 8597 raft_consensus.cc:303] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.484012 8597 raft_consensus.cc:329] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:10.484261 8597 raft_consensus.cc:604] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9a4e4b50a948dca456e1e9a3669b19, State: Initialized, Role: FOLLOWER I0506 07:57:10.493597 8597 consensus_queue.cc:227] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.500141 8573 heartbeater.cc:467] Master 127.0.196.126:52549 was elected leader, sending a full tablet report... I0506 07:57:10.502080 8597 ts_tablet_manager.cc:1073] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Time spent starting tablet: real 0.039s user 0.021s sys 0.001s I0506 07:57:10.484755 8408 heartbeater.cc:467] Master 127.0.196.126:52549 was elected leader, sending a full tablet report... I0506 07:57:10.639374 8595 ts_tablet_manager.cc:1073] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Time spent starting tablet: real 0.188s user 0.022s sys 0.093s W0506 07:57:10.541613 8574 tablet.cc:1678] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Can't schedule compaction. Clean time has not been advanced past its initial value. W0506 07:57:10.701758 8410 tablet.cc:1678] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0506 07:57:10.551157 8284 tablet.cc:1678] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:10.754017 8600 raft_consensus.cc:435] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0506 07:57:10.754349 8600 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Snoozing failure detection for 1.754s (starting election) I0506 07:57:10.754679 8600 raft_consensus.cc:457] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.756049 8600 leader_election.cc:230] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 069f235f3a0a42e4a371cb34d113e46d I0506 07:57:10.756620 8600 leader_election.cc:230] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer ca9a4e4b50a948dca456e1e9a3669b19 I0506 07:57:10.775146 8634 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.757481 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.66:32803 0506 07:57:10.770489 (+ 13008us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.770535 (+ 46us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.770682 (+ 147us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.772294 (+ 1612us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.772320 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.772381 (+ 61us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.772548 (+ 167us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.772568 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.772663 (+ 95us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.772773 (+ 110us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.773616 (+ 843us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.773645 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.773670 (+ 25us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.773797 (+ 127us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.773928 (+ 131us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":12793,"thread_start_us":13266,"threads_started":1} I0506 07:57:10.775166 8590 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.757579 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:39772 0506 07:57:10.757963 (+ 384us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.757987 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.770644 (+ 12657us) server_negotiation.cc:373] Connection header received 0506 07:57:10.770891 (+ 247us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.770919 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.771024 (+ 105us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.771148 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.773230 (+ 2082us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.773257 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.773282 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.773326 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.773388 (+ 62us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.773449 (+ 61us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.773558 (+ 109us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.773903 (+ 345us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.773992 (+ 89us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":165} I0506 07:57:10.776481 8365 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "b1717e9a2c794b599c375e90a9ded6a6" candidate_uuid: "2c3fe82a8554492ab39c0826fb959f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069f235f3a0a42e4a371cb34d113e46d" is_pre_election: true I0506 07:57:10.777149 8365 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Snoozing failure detection for 2.163s (vote granted) I0506 07:57:10.777433 8365 raft_consensus.cc:2219] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3fe82a8554492ab39c0826fb959f6f in term 0. I0506 07:57:10.779063 7965 leader_election.cc:362] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 pre-election: Vote granted by peer 069f235f3a0a42e4a371cb34d113e46d I0506 07:57:10.779356 7965 leader_election.cc:257] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0506 07:57:10.779932 8600 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Snoozing failure detection for 1.907s (election complete) I0506 07:57:10.780174 8600 raft_consensus.cc:2511] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0506 07:57:10.780395 8600 raft_consensus.cc:435] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0506 07:57:10.780622 8600 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Snoozing failure detection for 2.198s (starting election) I0506 07:57:10.780838 8600 raft_consensus.cc:2756] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:10.781888 8593 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.771476 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:60046 0506 07:57:10.771804 (+ 328us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.771827 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.778425 (+ 6598us) server_negotiation.cc:373] Connection header received 0506 07:57:10.778790 (+ 365us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.778817 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.778921 (+ 104us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.779047 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.779858 (+ 811us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.779887 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.779911 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.779956 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.780015 (+ 59us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.780073 (+ 58us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.780171 (+ 98us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.780607 (+ 436us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.780722 (+ 115us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} I0506 07:57:10.785394 8600 raft_consensus.cc:457] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.786464 8600 leader_election.cc:230] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 election: Requesting vote from peer 069f235f3a0a42e4a371cb34d113e46d I0506 07:57:10.786875 8600 leader_election.cc:230] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 election: Requesting vote from peer ca9a4e4b50a948dca456e1e9a3669b19 I0506 07:57:10.787648 8365 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "b1717e9a2c794b599c375e90a9ded6a6" candidate_uuid: "2c3fe82a8554492ab39c0826fb959f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069f235f3a0a42e4a371cb34d113e46d" I0506 07:57:10.788064 8365 raft_consensus.cc:2756] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:10.788336 8365 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Snoozing failure detection for 2.085s (vote granted) I0506 07:57:10.793335 8365 raft_consensus.cc:2219] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c3fe82a8554492ab39c0826fb959f6f in term 1. I0506 07:57:10.794085 7965 leader_election.cc:362] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 election: Vote granted by peer 069f235f3a0a42e4a371cb34d113e46d I0506 07:57:10.794327 7965 leader_election.cc:257] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:10.795601 8600 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f: Snoozing failure detection for 1.867s (election complete) I0506 07:57:10.795871 8600 raft_consensus.cc:2511] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:10.797634 8635 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.771403 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.67:43040 0506 07:57:10.778306 (+ 6903us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:10.778350 (+ 44us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:10.778555 (+ 205us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:10.779269 (+ 714us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:10.779295 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:10.779374 (+ 79us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:10.779463 (+ 89us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0506 07:57:10.779482 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:10.779516 (+ 34us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0506 07:57:10.779617 (+ 101us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:10.780324 (+ 707us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:10.780351 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:10.780375 (+ 24us) client_negotiation.cc:695] Sending connection context 0506 07:57:10.782373 (+ 1998us) client_negotiation.cc:225] Negotiation successful 0506 07:57:10.782501 (+ 128us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2175,"thread_start_us":1326,"threads_started":1} I0506 07:57:10.798972 8600 raft_consensus.cc:571] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [term 1 LEADER]: Becoming Leader. State: Replica: 2c3fe82a8554492ab39c0826fb959f6f, State: Running, Role: LEADER I0506 07:57:10.799163 8530 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "b1717e9a2c794b599c375e90a9ded6a6" candidate_uuid: "2c3fe82a8554492ab39c0826fb959f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" is_pre_election: true I0506 07:57:10.799474 8529 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "b1717e9a2c794b599c375e90a9ded6a6" candidate_uuid: "2c3fe82a8554492ab39c0826fb959f6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" I0506 07:57:10.800014 8530 raft_consensus.cc:2704] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Snoozing failure detection for 2.123s (vote granted) I0506 07:57:10.799543 8600 consensus_queue.cc:204] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } } I0506 07:57:10.800312 8530 raft_consensus.cc:2219] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c3fe82a8554492ab39c0826fb959f6f in term 0. I0506 07:57:10.800848 8529 raft_consensus.cc:2194] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c3fe82a8554492ab39c0826fb959f6f for term 1 because replica is already servicing an update from a current leader or another vote. I0506 07:57:10.801393 7965 leader_election.cc:362] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 pre-election: Vote granted by peer ca9a4e4b50a948dca456e1e9a3669b19 I0506 07:57:10.802116 7965 leader_election.cc:376] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [CANDIDATE]: Term 1 election: Vote denied by peer ca9a4e4b50a948dca456e1e9a3669b19. Message: Service unavailable: T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c3fe82a8554492ab39c0826fb959f6f for term 1 because replica is already servicing an update from a current leader or another vote. I0506 07:57:10.809276 7832 catalog_manager.cc:3716] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f reported cstate change: term changed from 0 to 1, leader changed from to 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65). New cstate: current_term: 1 leader_uuid: "2c3fe82a8554492ab39c0826fb959f6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fe82a8554492ab39c0826fb959f6f" member_type: VOTER last_known_addr { host: "127.0.196.65" port: 39558 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 } health_report { overall_health: UNKNOWN } } } I0506 07:57:10.868887 8588 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:10.863550 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:56121 0506 07:57:10.863829 (+ 279us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:10.863853 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:10.865409 (+ 1556us) server_negotiation.cc:373] Connection header received 0506 07:57:10.865749 (+ 340us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:10.865776 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:10.865873 (+ 97us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:10.865969 (+ 96us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:10.867108 (+ 1139us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:10.867135 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:10.867157 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:10.867197 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:10.867254 (+ 57us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:10.867310 (+ 56us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:10.867413 (+ 103us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:10.867729 (+ 316us) server_negotiation.cc:264] Negotiation successful 0506 07:57:10.867819 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":78} I0506 07:57:10.879936 8529 raft_consensus.cc:2756] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:10.880892 8365 raft_consensus.cc:1057] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 1 FOLLOWER]: Refusing update from remote peer 2c3fe82a8554492ab39c0826fb959f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:10.882225 8600 consensus_queue.cc:963] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "069f235f3a0a42e4a371cb34d113e46d" member_type: VOTER last_known_addr { host: "127.0.196.66" port: 32803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:10.884730 8529 raft_consensus.cc:1057] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 1 FOLLOWER]: Refusing update from remote peer 2c3fe82a8554492ab39c0826fb959f6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:10.886157 8600 consensus_queue.cc:963] T b1717e9a2c794b599c375e90a9ded6a6 P 2c3fe82a8554492ab39c0826fb959f6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9a4e4b50a948dca456e1e9a3669b19" member_type: VOTER last_known_addr { host: "127.0.196.67" port: 43040 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:10.963016 7832 catalog_manager.cc:1713] Servicing DeleteTable request from {username='jenkins'} at 127.0.196.126:34123: table { table_name: "test-table" } I0506 07:57:10.968952 7832 catalog_manager.cc:3927] T 00000000000000000000000000000000 P a484ae26c7c442229ecfb87426210e16: Sending DeleteTablet for 3 replicas of tablet b1717e9a2c794b599c375e90a9ded6a6 I0506 07:57:10.969288 7832 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1717e9a2c794b599c375e90a9ded6a6 on 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65:39558) (Table deleted at 2018-05-06 07:57:10 PDT) I0506 07:57:10.969770 7832 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1717e9a2c794b599c375e90a9ded6a6 on 069f235f3a0a42e4a371cb34d113e46d (127.0.196.66:32803) (Table deleted at 2018-05-06 07:57:10 PDT) I0506 07:57:10.970168 7832 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1717e9a2c794b599c375e90a9ded6a6 on ca9a4e4b50a948dca456e1e9a3669b19 (127.0.196.67:43040) (Table deleted at 2018-05-06 07:57:10 PDT) I0506 07:57:10.970943 8345 tablet_service.cc:801] Processing DeleteTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:57:10 PDT) from {username='jenkins'} at 127.0.196.66:43282 I0506 07:57:10.971300 8133 tablet_service.cc:801] Processing DeleteTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:57:10 PDT) from {username='jenkins'} at 127.0.196.65:56118 I0506 07:57:10.972743 8496 tablet_service.cc:801] Processing DeleteTablet for tablet b1717e9a2c794b599c375e90a9ded6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-05-06 07:57:10 PDT) from {username='jenkins'} at 127.0.196.67:38281 I0506 07:57:10.977201 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 7919 I0506 07:57:10.979115 8655 tablet_replica.cc:261] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: stopping tablet replica I0506 07:57:10.979440 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op CompactRowSetsOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.980185 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op MinorDeltaCompactionOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.980437 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op MajorDeltaCompactionOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.980669 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op UndoDeltaBlockGCOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.980971 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op FlushMRSOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.981195 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op FlushDeltaMemStoresOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.981415 8655 maintenance_manager.cc:237] P 069f235f3a0a42e4a371cb34d113e46d: Unregistered op LogGCOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.981664 8655 raft_consensus.cc:2016] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 1 FOLLOWER]: Raft consensus shutting down. I0506 07:57:10.982055 8655 raft_consensus.cc:2043] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:57:10.983038 8657 tablet_replica.cc:261] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: stopping tablet replica I0506 07:57:10.983326 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op CompactRowSetsOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.983542 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op MinorDeltaCompactionOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.983800 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op MajorDeltaCompactionOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.984066 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op UndoDeltaBlockGCOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.984330 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op FlushMRSOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.984529 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op FlushDeltaMemStoresOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.984726 8657 maintenance_manager.cc:237] P ca9a4e4b50a948dca456e1e9a3669b19: Unregistered op LogGCOp(b1717e9a2c794b599c375e90a9ded6a6) I0506 07:57:10.984949 8657 raft_consensus.cc:2016] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 1 FOLLOWER]: Raft consensus shutting down. I0506 07:57:10.985265 8657 raft_consensus.cc:2043] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19 [term 1 FOLLOWER]: Raft consensus is shut down! I0506 07:57:10.986377 8655 ts_tablet_manager.cc:1369] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:57:10.990736 8657 ts_tablet_manager.cc:1369] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Deleting tablet data with delete state TABLET_DATA_DELETED I0506 07:57:10.996951 8655 ts_tablet_manager.cc:1382] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:57:10.997251 8655 log.cc:981] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-1/wal/wals/b1717e9a2c794b599c375e90a9ded6a6 I0506 07:57:10.998029 8655 ts_tablet_manager.cc:1402] T b1717e9a2c794b599c375e90a9ded6a6 P 069f235f3a0a42e4a371cb34d113e46d: Deleting consensus metadata I0506 07:57:11.000572 7809 catalog_manager.cc:3021] TS 069f235f3a0a42e4a371cb34d113e46d (127.0.196.66:32803): tablet b1717e9a2c794b599c375e90a9ded6a6 (table test-table [id=de69b0a907d54bd4b4f184cb25929755]) successfully deleted I0506 07:57:11.008649 8657 ts_tablet_manager.cc:1382] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0506 07:57:11.008971 8657 log.cc:981] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1525618592290500-785/minicluster-data/ts-2/wal/wals/b1717e9a2c794b599c375e90a9ded6a6 I0506 07:57:11.009676 8657 ts_tablet_manager.cc:1402] T b1717e9a2c794b599c375e90a9ded6a6 P ca9a4e4b50a948dca456e1e9a3669b19: Deleting consensus metadata I0506 07:57:11.012249 7809 catalog_manager.cc:3021] TS ca9a4e4b50a948dca456e1e9a3669b19 (127.0.196.67:43040): tablet b1717e9a2c794b599c375e90a9ded6a6 (table test-table [id=de69b0a907d54bd4b4f184cb25929755]) successfully deleted W0506 07:57:11.103019 7811 catalog_manager.cc:2785] TS 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65:39558): Delete Tablet RPC failed for tablet b1717e9a2c794b599c375e90a9ded6a6: Network error: recv got EOF from 127.0.196.65:39558 (error 108) I0506 07:57:11.103303 7811 catalog_manager.cc:2821] Scheduling retry of b1717e9a2c794b599c375e90a9ded6a6 Delete Tablet RPC for TS=2c3fe82a8554492ab39c0826fb959f6f with a delay of 32 ms (attempt = 1) I0506 07:57:11.103955 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 8283 I0506 07:57:11.136886 7811 catalog_manager.cc:3043] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet b1717e9a2c794b599c375e90a9ded6a6 on 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65:39558) (Table deleted at 2018-05-06 07:57:10 PDT) I0506 07:57:11.139727 8658 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:11.137478 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.65:39558 0506 07:57:11.138964 (+ 1486us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:11.139166 (+ 202us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.65:39558: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":1218,"thread_start_us":933,"threads_started":1} W0506 07:57:11.140066 7811 catalog_manager.cc:2785] TS 2c3fe82a8554492ab39c0826fb959f6f (127.0.196.65:39558): Delete Tablet RPC failed for tablet b1717e9a2c794b599c375e90a9ded6a6: Network error: Client connection negotiation failed: client connection to 127.0.196.65:39558: connect: Connection refused (error 111) I0506 07:57:11.140312 7811 catalog_manager.cc:2821] Scheduling retry of b1717e9a2c794b599c375e90a9ded6a6 Delete Tablet RPC for TS=2c3fe82a8554492ab39c0826fb959f6f with a delay of 79 ms (attempt = 2) I0506 07:57:11.162762 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 8409 I0506 07:57:11.211129 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 7757 [ OK ] SecurityITest.TestDisableAuthenticationEncryption (5234 ms) [ RUN ] SecurityITest.TestWorldReadableKeytab [ OK ] SecurityITest.TestWorldReadableKeytab (241 ms) [ RUN ] SecurityITest.TestWorldReadablePrivateKey [ OK ] SecurityITest.TestWorldReadablePrivateKey (272 ms) [----------] 9 tests from SecurityITest (39352 ms total) [----------] 12 tests from AuthTokenIssuingTest [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): setting up network... May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): skipping unrecognized local address family 17 May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): skipping unrecognized local address family 17 May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): set up 1 sockets May 06 07:57:11 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): commencing operation krb5kdc: starting... W0506 07:57:11.960880 785 mini_kdc.cc:117] Time spent starting KDC: real 0.156s user 0.004s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:12 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:12.153117 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:55089 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:12.437378 8701 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:12.437945 8701 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:12.438381 8701 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:12.459955 8701 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:12.460224 8701 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:12.460417 8701 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:12.460608 8701 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:12.460856 8701 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:12.461093 8701 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:12.520309 8701 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:55089 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:12.540125 8701 master_main.cc:73] Initializing master server... I0506 07:57:12.543421 8701 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 211678us May 06 07:57:12 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:12.549921 8701 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:12.568183 8701 fs_manager.cc:260] Metadata directory not provided I0506 07:57:12.568403 8701 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:12.568801 8701 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:12.569277 8701 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:12.591085 8701 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "61fad8855e5847539d463594393f682c" format_stamp: "Formatted at 2018-05-06 14:57:12 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:12.592748 8701 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "61fad8855e5847539d463594393f682c" format_stamp: "Formatted at 2018-05-06 14:57:12 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:12.618249 8701 fs_manager.cc:495] Time spent creating directory manager: real 0.025s user 0.012s sys 0.007s I0506 07:57:12.620182 8701 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:12.621692 8701 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:12.648644 8701 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.004s I0506 07:57:12.648918 8701 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/wal uuid: "61fad8855e5847539d463594393f682c" format_stamp: "Formatted at 2018-05-06 14:57:12 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:12.649345 8701 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:13.016536 8701 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:13.026654 8701 master_main.cc:76] Starting Master server... I0506 07:57:13.348978 8701 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:55089 I0506 07:57:13.350430 8701 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:13.350607 8701 webserver.cc:184] Document root disabled I0506 07:57:13.358907 8701 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:49462/ I0506 07:57:13.359988 8701 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:13.366152 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 8701 I0506 07:57:13.394237 8766 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:13.428429 8766 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Bootstrap starting. May 06 07:57:13 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM I0506 07:57:13.431510 8766 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: No blocks or log segments found. Creating new log. I0506 07:57:13.442157 8766 log.cc:520] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Log is configured to *not* fsync() on all Append() calls I0506 07:57:13.451799 8768 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:13.369062 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:47262 0506 07:57:13.385576 (+ 16514us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:13.385654 (+ 78us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:13.385757 (+ 103us) server_negotiation.cc:373] Connection header received 0506 07:57:13.419776 (+ 34019us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:13.419826 (+ 50us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:13.420121 (+ 295us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:13.420457 (+ 336us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:13.422267 (+ 1810us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:13.423088 (+ 821us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:13.425049 (+ 1961us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:13.427360 (+ 2311us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:13.427583 (+ 223us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:13.432587 (+ 5004us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:13.432689 (+ 102us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:13.432724 (+ 35us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:13.432814 (+ 90us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:13.442714 (+ 9900us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:13.445099 (+ 2385us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:13.445127 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:13.445146 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:13.445839 (+ 693us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:13.447038 (+ 1199us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:13.447062 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:13.447079 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:13.447704 (+ 625us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:13.448278 (+ 574us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:13.449406 (+ 1128us) server_negotiation.cc:264] Negotiation successful 0506 07:57:13.449807 (+ 401us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":16007,"thread_start_us":8944,"threads_started":1} I0506 07:57:13.463160 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:13.468189 8766 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: No bootstrap required, opened a new log I0506 07:57:13.501245 8766 raft_consensus.cc:303] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } I0506 07:57:13.505360 8766 raft_consensus.cc:329] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:13.505877 8766 raft_consensus.cc:604] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61fad8855e5847539d463594393f682c, State: Initialized, Role: FOLLOWER I0506 07:57:13.506923 8766 consensus_queue.cc:227] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } I0506 07:57:13.508205 8766 raft_consensus.cc:343] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:13.508605 8766 raft_consensus.cc:435] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:13.508910 8766 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Snoozing failure detection for 1.772s (starting election) I0506 07:57:13.509217 8766 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:13.513432 8766 raft_consensus.cc:457] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } I0506 07:57:13.514075 8766 leader_election.cc:257] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:13.521721 8775 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Snoozing failure detection for 1.830s (election complete) I0506 07:57:13.524116 8775 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:13.533885 8775 raft_consensus.cc:571] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [term 1 LEADER]: Becoming Leader. State: Replica: 61fad8855e5847539d463594393f682c, State: Running, Role: LEADER I0506 07:57:13.534428 8775 consensus_queue.cc:204] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } I0506 07:57:13.542135 8776 sys_catalog.cc:335] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61fad8855e5847539d463594393f682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } } I0506 07:57:13.542562 8776 sys_catalog.cc:338] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: This master's current role is: LEADER I0506 07:57:13.557561 8768 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:13.509368 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:48391 0506 07:57:13.509632 (+ 264us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:13.509675 (+ 43us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:13.516531 (+ 6856us) server_negotiation.cc:373] Connection header received 0506 07:57:13.516811 (+ 280us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:13.516837 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:13.516956 (+ 119us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:13.517080 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:13.525075 (+ 7995us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:13.525973 (+ 898us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:13.527900 (+ 1927us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:13.529378 (+ 1478us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:13.529540 (+ 162us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:13.533341 (+ 3801us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:13.533409 (+ 68us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:13.533441 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:13.533521 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:13.540601 (+ 7080us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:13.549041 (+ 8440us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:13.549068 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:13.549086 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:13.549768 (+ 682us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:13.551434 (+ 1666us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:13.551460 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:13.551478 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:13.552002 (+ 524us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:13.552500 (+ 498us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:13.555520 (+ 3020us) server_negotiation.cc:264] Negotiation successful 0506 07:57:13.555859 (+ 339us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":70} I0506 07:57:13.548408 8777 sys_catalog.cc:335] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61fad8855e5847539d463594393f682c. Latest consensus state: current_term: 1 leader_uuid: "61fad8855e5847539d463594393f682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } } I0506 07:57:13.732648 8777 sys_catalog.cc:338] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: This master's current role is: LEADER I0506 07:57:13.544855 8766 sys_catalog.cc:335] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "61fad8855e5847539d463594393f682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61fad8855e5847539d463594393f682c" member_type: VOTER } } I0506 07:57:13.744410 8766 sys_catalog.cc:338] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: This master's current role is: LEADER I0506 07:57:13.745733 8766 sys_catalog.cc:422] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:13.760098 8779 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:13.760654 8701 master_main.cc:79] Master server successfully started. I0506 07:57:13.775380 8779 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:57:13.784251 8722 tablet.cc:1678] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:13.851325 8779 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:13.853782 8779 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:13.888736 8779 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 61fad8855e5847539d463594393f682c: Generated new TSK 0 I0506 07:57:13.909677 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 8701 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2179 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1 W0506 07:57:13.958386 8674 connection.cc:512] client connection to 127.0.0.1:55089 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:55089): Connection reset by peer (error 104) Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): setting up network... May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): skipping unrecognized local address family 17 May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): skipping unrecognized local address family 17 May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): set up 1 sockets May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): commencing operation krb5kdc: starting... W0506 07:57:14.158838 785 mini_kdc.cc:117] Time spent starting KDC: real 0.180s user 0.004s sys 0.012s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:14.295745 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:50414 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:14.467496 8795 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:14.468035 8795 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:14.468453 8795 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:14.477982 8795 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:14.478142 8795 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:14.478266 8795 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:14.478382 8795 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:14.478515 8795 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:14.478677 8795 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:14.499862 8795 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:50414 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:14.509975 8795 master_main.cc:73] Initializing master server... I0506 07:57:14.511996 8795 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 212678us May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:14.518227 8795 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:14.524221 8795 fs_manager.cc:260] Metadata directory not provided I0506 07:57:14.524374 8795 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:14.524607 8795 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:14.524772 8795 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:14.534906 8795 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" format_stamp: "Formatted at 2018-05-06 14:57:14 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:14.535614 8795 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" format_stamp: "Formatted at 2018-05-06 14:57:14 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:14.547754 8795 fs_manager.cc:495] Time spent creating directory manager: real 0.012s user 0.008s sys 0.003s I0506 07:57:14.548943 8795 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:14.550140 8795 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:14.559798 8795 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0506 07:57:14.559923 8795 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/wal uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" format_stamp: "Formatted at 2018-05-06 14:57:14 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:14.560153 8795 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:14.771600 8795 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:14.774807 8795 master_main.cc:76] Starting Master server... I0506 07:57:14.883846 8795 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:50414 I0506 07:57:14.884763 8795 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:14.884873 8795 webserver.cc:184] Document root disabled I0506 07:57:14.888298 8795 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:52274/ I0506 07:57:14.889058 8795 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:14.891047 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 8795 I0506 07:57:14.897378 8857 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed May 06 07:57:14 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM I0506 07:57:14.930547 8857 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: Bootstrap starting. I0506 07:57:14.933290 8857 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: No blocks or log segments found. Creating new log. I0506 07:57:14.939692 8859 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:14.893121 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:47837 0506 07:57:14.897100 (+ 3979us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:14.897135 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:14.897201 (+ 66us) server_negotiation.cc:373] Connection header received 0506 07:57:14.907613 (+ 10412us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:14.907658 (+ 45us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:14.907984 (+ 326us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:14.911009 (+ 3025us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:14.911404 (+ 395us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:14.912184 (+ 780us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:14.914290 (+ 2106us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:14.916566 (+ 2276us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:14.919911 (+ 3345us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:14.921539 (+ 1628us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:14.921584 (+ 45us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:14.921620 (+ 36us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:14.921751 (+ 131us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:14.929769 (+ 8018us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:14.931553 (+ 1784us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:14.931581 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:14.931600 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:14.932348 (+ 748us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:14.932923 (+ 575us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:14.932946 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:14.932962 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:14.933764 (+ 802us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:14.934169 (+ 405us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:14.934857 (+ 688us) server_negotiation.cc:264] Negotiation successful 0506 07:57:14.935359 (+ 502us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3589,"thread_start_us":3007,"threads_started":1} I0506 07:57:14.942772 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:14.970283 8859 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:14.955430 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:34240 0506 07:57:14.955649 (+ 219us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:14.955683 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:14.955710 (+ 27us) server_negotiation.cc:373] Connection header received 0506 07:57:14.955777 (+ 67us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:14.955791 (+ 14us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:14.955868 (+ 77us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:14.955969 (+ 101us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:14.957444 (+ 1475us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:14.957792 (+ 348us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:14.958923 (+ 1131us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:14.959786 (+ 863us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:14.959946 (+ 160us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:14.961834 (+ 1888us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:14.961851 (+ 17us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:14.961863 (+ 12us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:14.961905 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:14.965672 (+ 3767us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:14.966708 (+ 1036us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:14.966725 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:14.966734 (+ 9us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:14.967123 (+ 389us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:14.967498 (+ 375us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:14.967514 (+ 16us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:14.967523 (+ 9us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:14.967847 (+ 324us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:14.968115 (+ 268us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:14.968568 (+ 453us) server_negotiation.cc:264] Negotiation successful 0506 07:57:14.969032 (+ 464us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":70} I0506 07:57:14.940026 8857 log.cc:520] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: Log is configured to *not* fsync() on all Append() calls I0506 07:57:15.026049 8857 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: No bootstrap required, opened a new log I0506 07:57:15.037744 8857 raft_consensus.cc:303] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } I0506 07:57:15.038173 8857 raft_consensus.cc:329] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:15.038390 8857 raft_consensus.cc:604] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88eef7065d5342f8b4a7f63f89bfb8e7, State: Initialized, Role: FOLLOWER I0506 07:57:15.039057 8857 consensus_queue.cc:227] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } I0506 07:57:15.039572 8857 raft_consensus.cc:343] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:15.039855 8857 raft_consensus.cc:435] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:15.040104 8857 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: Snoozing failure detection for 2.051s (starting election) I0506 07:57:15.040351 8857 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:15.043452 8857 raft_consensus.cc:457] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } I0506 07:57:15.044206 8857 leader_election.cc:257] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:15.047226 8867 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: Snoozing failure detection for 2.032s (election complete) I0506 07:57:15.047698 8867 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:15.053439 8867 raft_consensus.cc:571] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [term 1 LEADER]: Becoming Leader. State: Replica: 88eef7065d5342f8b4a7f63f89bfb8e7, State: Running, Role: LEADER I0506 07:57:15.054157 8867 consensus_queue.cc:204] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } I0506 07:57:15.063125 8868 sys_catalog.cc:335] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } } I0506 07:57:15.063576 8868 sys_catalog.cc:338] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:15.063279 8857 sys_catalog.cc:335] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } } I0506 07:57:15.063458 8869 sys_catalog.cc:335] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88eef7065d5342f8b4a7f63f89bfb8e7. Latest consensus state: current_term: 1 leader_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88eef7065d5342f8b4a7f63f89bfb8e7" member_type: VOTER } } I0506 07:57:15.063838 8857 sys_catalog.cc:338] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:15.063984 8869 sys_catalog.cc:338] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:15.065511 8857 sys_catalog.cc:422] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:15.072443 8873 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:15.073746 8795 master_main.cc:79] Master server successfully started. I0506 07:57:15.073155 8872 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:15.080757 8872 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:15.179690 8872 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:15.182304 8872 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:15.212324 8872 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 88eef7065d5342f8b4a7f63f89bfb8e7: Generated new TSK 0 I0506 07:57:15.233963 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 8795 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1316 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): setting up network... May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): skipping unrecognized local address family 17 May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): skipping unrecognized local address family 17 May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): set up 1 sockets May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): commencing operation krb5kdc: starting... W0506 07:57:15.439039 785 mini_kdc.cc:117] Time spent starting KDC: real 0.150s user 0.003s sys 0.012s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:15.576437 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:56278 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:15.775352 8965 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:15.775878 8965 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:15.776281 8965 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:15.795387 8965 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:15.795699 8965 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:15.795899 8965 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:15.796089 8965 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:15.796295 8965 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:15.796516 8965 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:15.841778 8965 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:56278 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:15.868898 8965 master_main.cc:73] Initializing master server... I0506 07:57:15.872079 8965 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 213178us May 06 07:57:15 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[8931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:15.892380 8965 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:15.921072 8965 fs_manager.cc:260] Metadata directory not provided I0506 07:57:15.921262 8965 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:15.921552 8965 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:15.921756 8965 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:15.937247 8965 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "4a111e5a58d747b5a196ec516300777a" format_stamp: "Formatted at 2018-05-06 14:57:15 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:15.938474 8965 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "4a111e5a58d747b5a196ec516300777a" format_stamp: "Formatted at 2018-05-06 14:57:15 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:15.958204 8965 fs_manager.cc:495] Time spent creating directory manager: real 0.019s user 0.012s sys 0.003s I0506 07:57:15.960155 8965 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:15.961668 8965 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:15.975900 8965 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0506 07:57:15.976114 8965 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/wal uuid: "4a111e5a58d747b5a196ec516300777a" format_stamp: "Formatted at 2018-05-06 14:57:15 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:15.976480 8965 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:16.289052 8965 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:16.296911 8965 master_main.cc:76] Starting Master server... I0506 07:57:16.577224 8965 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:56278 I0506 07:57:16.578547 8965 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:16.578743 8965 webserver.cc:184] Document root disabled I0506 07:57:16.585491 8965 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:43590/ I0506 07:57:16.586552 8965 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:16.592238 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 8965 I0506 07:57:16.601059 9137 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:16.647842 9137 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: Bootstrap starting. I0506 07:57:16.653442 9143 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:16.598559 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:45203 0506 07:57:16.622809 (+ 24250us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:16.622851 (+ 42us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:16.622932 (+ 81us) server_negotiation.cc:373] Connection header received 0506 07:57:16.629593 (+ 6661us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:16.629643 (+ 50us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:16.629959 (+ 316us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:16.631729 (+ 1770us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:16.634217 (+ 2488us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:16.636580 (+ 2363us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:16.641485 (+ 4905us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:16.644263 (+ 2778us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:16.644480 (+ 217us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:16.648389 (+ 3909us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:16.648468 (+ 79us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:16.648499 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:16.649114 (+ 615us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:16.649213 (+ 99us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:16.649297 (+ 84us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:16.649681 (+ 384us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:16.650983 (+ 1302us) server_negotiation.cc:264] Negotiation successful 0506 07:57:16.651121 (+ 138us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":23850,"thread_start_us":22604,"threads_started":1} I0506 07:57:16.656976 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:16.661398 9137 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: No blocks or log segments found. Creating new log. I0506 07:57:16.665350 9137 log.cc:520] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: Log is configured to *not* fsync() on all Append() calls I0506 07:57:16.675115 9137 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: No bootstrap required, opened a new log I0506 07:57:16.707401 9137 raft_consensus.cc:303] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } I0506 07:57:16.708559 9137 raft_consensus.cc:329] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:16.709106 9137 raft_consensus.cc:604] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a111e5a58d747b5a196ec516300777a, State: Initialized, Role: FOLLOWER I0506 07:57:16.710286 9137 consensus_queue.cc:227] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } I0506 07:57:16.712095 9137 raft_consensus.cc:343] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:16.712716 9137 raft_consensus.cc:435] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:16.713351 9137 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: Snoozing failure detection for 2.098s (starting election) I0506 07:57:16.713935 9137 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:16.715638 9143 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:16.691258 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:49006 0506 07:57:16.691538 (+ 280us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:16.691560 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:16.705508 (+ 13948us) server_negotiation.cc:373] Connection header received 0506 07:57:16.705798 (+ 290us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:16.705822 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:16.705938 (+ 116us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:16.706065 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:16.707083 (+ 1018us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:16.707661 (+ 578us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:16.709479 (+ 1818us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:16.711011 (+ 1532us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:16.711186 (+ 175us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:16.712623 (+ 1437us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:16.712697 (+ 74us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:16.712722 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:16.712848 (+ 126us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:16.712920 (+ 72us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:16.712984 (+ 64us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:16.713280 (+ 296us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:16.714078 (+ 798us) server_negotiation.cc:264] Negotiation successful 0506 07:57:16.714168 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":82} I0506 07:57:16.718739 9137 raft_consensus.cc:457] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } I0506 07:57:16.825680 9137 leader_election.cc:257] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:16.836940 9181 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: Snoozing failure detection for 2.244s (election complete) I0506 07:57:16.838667 9181 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:16.856726 9181 raft_consensus.cc:571] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [term 1 LEADER]: Becoming Leader. State: Replica: 4a111e5a58d747b5a196ec516300777a, State: Running, Role: LEADER I0506 07:57:16.857266 9181 consensus_queue.cc:204] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } I0506 07:57:16.866577 9186 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a111e5a58d747b5a196ec516300777a. Latest consensus state: current_term: 1 leader_uuid: "4a111e5a58d747b5a196ec516300777a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } } I0506 07:57:16.867055 9186 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: This master's current role is: LEADER I0506 07:57:16.868786 9137 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4a111e5a58d747b5a196ec516300777a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } } I0506 07:57:16.869228 9137 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: This master's current role is: LEADER I0506 07:57:16.871830 9185 sys_catalog.cc:335] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a111e5a58d747b5a196ec516300777a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a111e5a58d747b5a196ec516300777a" member_type: VOTER } } I0506 07:57:16.872231 9185 sys_catalog.cc:338] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: This master's current role is: LEADER I0506 07:57:16.872939 9137 sys_catalog.cc:422] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:16.880380 8965 master_main.cc:79] Master server successfully started. W0506 07:57:16.887866 9194 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:16.888108 9194 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:16.888339 9192 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:16.896508 9192 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:17.028507 9192 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:17.031009 9192 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:17.073829 9192 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 4a111e5a58d747b5a196ec516300777a: Generated new TSK 0 I0506 07:57:17.087338 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 8965 W0506 07:57:17.129148 8901 connection.cc:512] client connection to 127.0.0.1:56278 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:56278): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1859 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): setting up network... May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): skipping unrecognized local address family 17 May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): skipping unrecognized local address family 17 May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): set up 1 sockets May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): commencing operation krb5kdc: starting... W0506 07:57:17.350805 785 mini_kdc.cc:117] Time spent starting KDC: real 0.195s user 0.001s sys 0.018s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:17.543296 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:44441 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:17.771459 9312 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:17.773149 9312 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:17.773404 9312 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:17.789176 9312 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:17.789417 9312 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:17.789602 9312 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:17.789808 9312 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:17.790014 9312 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:17.790231 9312 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:17.845597 9312 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:44441 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:17.868568 9312 master_main.cc:73] Initializing master server... I0506 07:57:17.872859 9312 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 214178us May 06 07:57:17 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:17.895681 9312 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:17.909857 9312 fs_manager.cc:260] Metadata directory not provided I0506 07:57:17.910038 9312 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:17.910328 9312 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:17.910501 9312 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:17.935271 9312 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "360d62fa954a482493483dad975bdc84" format_stamp: "Formatted at 2018-05-06 14:57:17 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:17.938143 9312 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "360d62fa954a482493483dad975bdc84" format_stamp: "Formatted at 2018-05-06 14:57:17 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:17.959036 9312 fs_manager.cc:495] Time spent creating directory manager: real 0.020s user 0.009s sys 0.008s I0506 07:57:17.960764 9312 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:17.962153 9312 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:17.979882 9312 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.000s sys 0.005s I0506 07:57:17.980059 9312 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/wal uuid: "360d62fa954a482493483dad975bdc84" format_stamp: "Formatted at 2018-05-06 14:57:17 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:17.980403 9312 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:18.211813 9312 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:18.219274 9312 master_main.cc:76] Starting Master server... I0506 07:57:18.358760 9312 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:44441 I0506 07:57:18.359577 9312 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:18.359722 9312 webserver.cc:184] Document root disabled I0506 07:57:18.361232 9312 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:53693/ I0506 07:57:18.362150 9312 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:18.365495 9527 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:18.365968 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 9312 I0506 07:57:18.413722 9530 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:18.367586 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:49559 0506 07:57:18.372879 (+ 5293us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:18.373000 (+ 121us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:18.384007 (+ 11007us) server_negotiation.cc:373] Connection header received 0506 07:57:18.390618 (+ 6611us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:18.390689 (+ 71us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:18.390937 (+ 248us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:18.391221 (+ 284us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:18.392688 (+ 1467us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:18.393380 (+ 692us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:18.395534 (+ 2154us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:18.397749 (+ 2215us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:18.397935 (+ 186us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:18.401573 (+ 3638us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:18.401608 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:18.401663 (+ 55us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:18.402230 (+ 567us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:18.402320 (+ 90us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:18.402391 (+ 71us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:18.402501 (+ 110us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:18.411759 (+ 9258us) server_negotiation.cc:264] Negotiation successful 0506 07:57:18.411990 (+ 231us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4863,"thread_start_us":2903,"threads_started":1} I0506 07:57:18.416162 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:18.416332 9527 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: Bootstrap starting. I0506 07:57:18.419054 9527 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: No blocks or log segments found. Creating new log. I0506 07:57:18.420655 9527 log.cc:520] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: Log is configured to *not* fsync() on all Append() calls I0506 07:57:18.428189 9527 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: No bootstrap required, opened a new log I0506 07:57:18.445797 9527 raft_consensus.cc:303] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } I0506 07:57:18.446336 9527 raft_consensus.cc:329] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:18.446588 9527 raft_consensus.cc:604] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360d62fa954a482493483dad975bdc84, State: Initialized, Role: FOLLOWER I0506 07:57:18.448169 9527 consensus_queue.cc:227] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } I0506 07:57:18.449378 9527 raft_consensus.cc:343] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:18.449631 9527 raft_consensus.cc:435] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:18.449985 9527 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: Snoozing failure detection for 1.603s (starting election) I0506 07:57:18.450376 9527 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:18.457299 9527 raft_consensus.cc:457] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } I0506 07:57:18.457947 9527 leader_election.cc:257] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:18.463192 9530 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:18.447210 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:46214 0506 07:57:18.447507 (+ 297us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:18.447528 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:18.454183 (+ 6655us) server_negotiation.cc:373] Connection header received 0506 07:57:18.454421 (+ 238us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:18.454446 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:18.454558 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:18.454709 (+ 151us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:18.455712 (+ 1003us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:18.456211 (+ 499us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:18.458449 (+ 2238us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:18.459839 (+ 1390us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:18.460017 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:18.461068 (+ 1051us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:18.461094 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:18.461116 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:18.461166 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:18.461231 (+ 65us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:18.461297 (+ 66us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:18.461405 (+ 108us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:18.461690 (+ 285us) server_negotiation.cc:264] Negotiation successful 0506 07:57:18.461887 (+ 197us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":115} I0506 07:57:18.468219 9555 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: Snoozing failure detection for 1.618s (election complete) I0506 07:57:18.584920 9555 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:18.592895 9555 raft_consensus.cc:571] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [term 1 LEADER]: Becoming Leader. State: Replica: 360d62fa954a482493483dad975bdc84, State: Running, Role: LEADER I0506 07:57:18.593297 9555 consensus_queue.cc:204] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } I0506 07:57:18.602222 9577 sys_catalog.cc:335] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360d62fa954a482493483dad975bdc84. Latest consensus state: current_term: 1 leader_uuid: "360d62fa954a482493483dad975bdc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } } I0506 07:57:18.602708 9577 sys_catalog.cc:338] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: This master's current role is: LEADER I0506 07:57:18.604032 9527 sys_catalog.cc:335] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "360d62fa954a482493483dad975bdc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } } I0506 07:57:18.604459 9527 sys_catalog.cc:338] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: This master's current role is: LEADER I0506 07:57:18.604522 9561 sys_catalog.cc:335] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360d62fa954a482493483dad975bdc84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360d62fa954a482493483dad975bdc84" member_type: VOTER } } I0506 07:57:18.605095 9561 sys_catalog.cc:338] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: This master's current role is: LEADER I0506 07:57:18.605959 9527 sys_catalog.cc:422] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:18.608319 9579 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:18.619195 9312 master_main.cc:79] Master server successfully started. I0506 07:57:18.630451 9579 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:18.739894 9579 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:18.742332 9579 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:18.775535 9579 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 360d62fa954a482493483dad975bdc84: Generated new TSK 0 I0506 07:57:18.792876 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 9312 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1692 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): setting up network... May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): skipping unrecognized local address family 17 May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): skipping unrecognized local address family 17 May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): set up 1 sockets May 06 07:57:18 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): commencing operation krb5kdc: starting... W0506 07:57:19.000775 785 mini_kdc.cc:117] Time spent starting KDC: real 0.156s user 0.002s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:19 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:19.192914 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:38685 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:19.449828 9661 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:19.450258 9661 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:19.450646 9661 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:19.471415 9661 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:19.471714 9661 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:19.471912 9661 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:19.472105 9661 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:19.472316 9661 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:19.472579 9661 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:19.541281 9661 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:38685 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:19.554610 9661 master_main.cc:73] Initializing master server... I0506 07:57:19.558656 9661 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 215178us May 06 07:57:19 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[9606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:19.568691 9661 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:19.590709 9661 fs_manager.cc:260] Metadata directory not provided I0506 07:57:19.591003 9661 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:19.591351 9661 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:19.591579 9661 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:19.621317 9661 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "3ad382cb7726440288e6ca3e4ce3199d" format_stamp: "Formatted at 2018-05-06 14:57:19 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:19.623689 9661 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "3ad382cb7726440288e6ca3e4ce3199d" format_stamp: "Formatted at 2018-05-06 14:57:19 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:19.655916 9661 fs_manager.cc:495] Time spent creating directory manager: real 0.032s user 0.013s sys 0.007s I0506 07:57:19.657852 9661 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:19.658536 9661 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:19.680758 9661 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s I0506 07:57:19.680968 9661 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/wal uuid: "3ad382cb7726440288e6ca3e4ce3199d" format_stamp: "Formatted at 2018-05-06 14:57:19 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:19.681335 9661 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0506 07:57:20.253321 9661 thread.cc:554] rpc reactor (reactor) Time spent creating pthread: real 0.508s user 0.019s sys 0.106s W0506 07:57:20.253587 9661 thread.cc:521] rpc reactor (reactor) Time spent starting thread: real 0.509s user 0.019s sys 0.106s I0506 07:57:20.395834 9661 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:20.405184 9661 master_main.cc:76] Starting Master server... I0506 07:57:20.677366 9661 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:38685 I0506 07:57:20.678732 9661 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:20.678906 9661 webserver.cc:184] Document root disabled I0506 07:57:20.685055 9661 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:37318/ I0506 07:57:20.686134 9661 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:20.693615 9892 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:20.693965 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 9661 I0506 07:57:20.728926 9896 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:20.695959 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:52784 0506 07:57:20.711367 (+ 15408us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:20.711404 (+ 37us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:20.711470 (+ 66us) server_negotiation.cc:373] Connection header received 0506 07:57:20.723693 (+ 12223us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:20.723738 (+ 45us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:20.723957 (+ 219us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:20.724975 (+ 1018us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:20.725784 (+ 809us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:20.725816 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:20.725843 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:20.725890 (+ 47us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:20.725974 (+ 84us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:20.726042 (+ 68us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:20.727065 (+ 1023us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:20.727237 (+ 172us) server_negotiation.cc:264] Negotiation successful 0506 07:57:20.727364 (+ 127us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15088,"thread_start_us":15173,"threads_started":1} I0506 07:57:20.739478 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:20.786841 9896 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:20.769262 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:49364 0506 07:57:20.769580 (+ 318us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:20.769601 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:20.779918 (+ 10317us) server_negotiation.cc:373] Connection header received 0506 07:57:20.780534 (+ 616us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:20.780559 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:20.780690 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:20.780813 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:20.781656 (+ 843us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:20.781680 (+ 24us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:20.781702 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:20.781742 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:20.781807 (+ 65us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:20.781870 (+ 63us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:20.781963 (+ 93us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:20.785618 (+ 3655us) server_negotiation.cc:264] Negotiation successful 0506 07:57:20.785731 (+ 113us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":130} I0506 07:57:20.733525 9892 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Bootstrap starting. I0506 07:57:20.857118 9892 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: No blocks or log segments found. Creating new log. I0506 07:57:20.858943 9892 log.cc:520] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Log is configured to *not* fsync() on all Append() calls I0506 07:57:20.876935 9892 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: No bootstrap required, opened a new log I0506 07:57:20.928050 9892 raft_consensus.cc:303] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } I0506 07:57:20.930740 9892 raft_consensus.cc:329] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:20.931002 9892 raft_consensus.cc:604] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad382cb7726440288e6ca3e4ce3199d, State: Initialized, Role: FOLLOWER I0506 07:57:20.937394 9892 consensus_queue.cc:227] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } I0506 07:57:20.938221 9892 raft_consensus.cc:343] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:20.938519 9892 raft_consensus.cc:435] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:20.938854 9892 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Snoozing failure detection for 2.248s (starting election) I0506 07:57:20.939141 9892 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:20.943693 9892 raft_consensus.cc:457] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } I0506 07:57:20.946080 9892 leader_election.cc:257] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:20.957414 9944 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Snoozing failure detection for 1.937s (election complete) I0506 07:57:20.957783 9944 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:20.974548 9944 raft_consensus.cc:571] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [term 1 LEADER]: Becoming Leader. State: Replica: 3ad382cb7726440288e6ca3e4ce3199d, State: Running, Role: LEADER I0506 07:57:20.975122 9944 consensus_queue.cc:204] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } I0506 07:57:20.988982 9892 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "3ad382cb7726440288e6ca3e4ce3199d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } } I0506 07:57:20.989414 9892 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: This master's current role is: LEADER I0506 07:57:20.992522 9946 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ad382cb7726440288e6ca3e4ce3199d. Latest consensus state: current_term: 1 leader_uuid: "3ad382cb7726440288e6ca3e4ce3199d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } } I0506 07:57:20.992436 9947 sys_catalog.cc:335] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ad382cb7726440288e6ca3e4ce3199d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ad382cb7726440288e6ca3e4ce3199d" member_type: VOTER } } I0506 07:57:20.992957 9946 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: This master's current role is: LEADER I0506 07:57:20.992965 9947 sys_catalog.cc:338] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: This master's current role is: LEADER I0506 07:57:21.022477 9892 sys_catalog.cc:422] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:21.022785 9954 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:21.038247 9961 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:21.040069 9661 master_main.cc:79] Master server successfully started. I0506 07:57:21.053165 9954 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:57:21.168859 9797 tablet.cc:1678] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:21.448894 9954 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:21.449436 9954 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:21.510982 9954 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 3ad382cb7726440288e6ca3e4ce3199d: Generated new TSK 0 I0506 07:57:21.551476 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 9661 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2795 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): setting up network... May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): skipping unrecognized local address family 17 May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): skipping unrecognized local address family 17 May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): set up 1 sockets May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): commencing operation krb5kdc: starting... W0506 07:57:22.235272 785 mini_kdc.cc:117] Time spent starting KDC: real 0.582s user 0.002s sys 0.017s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. W0506 07:57:22.430398 785 mini_kdc.cc:250] Time spent creating user principal test-admin: real 0.195s user 0.001s sys 0.004s WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. W0506 07:57:22.603034 785 mini_kdc.cc:250] Time spent creating user principal test-user: real 0.172s user 0.000s sys 0.006s WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. W0506 07:57:22.764560 785 mini_kdc.cc:250] Time spent creating user principal joe-interloper: real 0.161s user 0.000s sys 0.007s May 06 07:57:22 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: W0506 07:57:22.928592 785 mini_kdc.cc:276] Time spent kinit for test-admin: real 0.164s user 0.001s sys 0.009s WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab. I0506 07:57:22.993046 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:41768 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:23.645673 10044 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:23.646162 10044 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0506 07:57:23.646605 10044 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:23.701481 10044 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:23.701802 10044 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:23.702011 10044 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:23.702845 10044 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:23.703151 10044 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:23.703413 10044 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.0.1 I0506 07:57:23.786484 10044 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:41768 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:23.821774 10044 master_main.cc:73] Initializing master server... I0506 07:57:23.824926 10044 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 217178us May 06 07:57:23 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:23.842545 10044 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0506 07:57:23.906023 10044 fs_manager.cc:260] Metadata directory not provided I0506 07:57:23.906270 10044 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:23.907285 10044 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:23.907531 10044 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:23.952648 10044 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" format_stamp: "Formatted at 2018-05-06 14:57:23 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:23.959704 10044 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" format_stamp: "Formatted at 2018-05-06 14:57:23 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:24.078595 10044 fs_manager.cc:495] Time spent creating directory manager: real 0.116s user 0.013s sys 0.006s I0506 07:57:24.080584 10044 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:24.081212 10044 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:24.165803 10044 fs_manager.cc:417] Time spent opening block manager: real 0.051s user 0.000s sys 0.018s I0506 07:57:24.166040 10044 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/wal uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" format_stamp: "Formatted at 2018-05-06 14:57:23 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:24.166828 10044 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:24.840386 10044 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:24.874908 10044 master_main.cc:76] Starting Master server... I0506 07:57:25.439255 10044 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:41768 I0506 07:57:25.441007 10044 webserver.cc:173] Starting webserver on 127.0.0.1:0 I0506 07:57:25.441191 10044 webserver.cc:184] Document root disabled I0506 07:57:25.456331 10044 webserver.cc:311] Webserver started. Bound to: http://127.0.0.1:52016/ I0506 07:57:25.458078 10044 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:25.463618 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 10044 I0506 07:57:25.480764 10147 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:25.590914 10147 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Bootstrap starting. I0506 07:57:25.599465 10147 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: No blocks or log segments found. Creating new log. I0506 07:57:25.607807 10147 log.cc:520] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Log is configured to *not* fsync() on all Append() calls I0506 07:57:25.591048 10149 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:25.465963 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:57544 0506 07:57:25.524145 (+ 58182us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:25.524199 (+ 54us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:25.524293 (+ 94us) server_negotiation.cc:373] Connection header received 0506 07:57:25.555597 (+ 31304us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:25.555674 (+ 77us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:25.556245 (+ 571us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:25.567956 (+ 11711us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:25.569752 (+ 1796us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:25.569801 (+ 49us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:25.569832 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:25.569899 (+ 67us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:25.569990 (+ 91us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:25.570071 (+ 81us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:25.581432 (+ 11361us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:25.581669 (+ 237us) server_negotiation.cc:264] Negotiation successful 0506 07:57:25.581926 (+ 257us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":48704,"thread_start_us":48277,"threads_started":1} I0506 07:57:25.994104 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:26.001626 10147 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: No bootstrap required, opened a new log I0506 07:57:26.069962 10149 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:26.040217 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51504 0506 07:57:26.040576 (+ 359us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:26.040599 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:26.057425 (+ 16826us) server_negotiation.cc:373] Connection header received 0506 07:57:26.058758 (+ 1333us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:26.058784 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:26.058899 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:26.059031 (+ 132us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:26.067528 (+ 8497us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:26.067557 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:26.067581 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:26.067659 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:26.067741 (+ 82us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:26.067808 (+ 67us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:26.067905 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:26.068328 (+ 423us) server_negotiation.cc:264] Negotiation successful 0506 07:57:26.068424 (+ 96us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":113} I0506 07:57:26.074309 10147 raft_consensus.cc:303] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } I0506 07:57:26.083796 10147 raft_consensus.cc:329] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:26.084051 10147 raft_consensus.cc:604] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9cc35a40d3041238401dfcb7f4a7fd7, State: Initialized, Role: FOLLOWER I0506 07:57:26.084938 10147 consensus_queue.cc:227] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } I0506 07:57:26.085953 10147 raft_consensus.cc:343] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:26.086252 10147 raft_consensus.cc:435] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:26.087560 10147 raft_consensus.cc:2704] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Snoozing failure detection for 1.926s (starting election) I0506 07:57:26.093489 10147 raft_consensus.cc:2756] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:26.123881 10147 raft_consensus.cc:457] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } I0506 07:57:26.125543 10147 leader_election.cc:257] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:26.147187 10168 raft_consensus.cc:2704] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Snoozing failure detection for 1.629s (election complete) I0506 07:57:26.156596 10168 raft_consensus.cc:2511] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:26.168952 10168 raft_consensus.cc:571] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [term 1 LEADER]: Becoming Leader. State: Replica: c9cc35a40d3041238401dfcb7f4a7fd7, State: Running, Role: LEADER I0506 07:57:26.169558 10168 consensus_queue.cc:204] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } I0506 07:57:26.188844 10168 sys_catalog.cc:335] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9cc35a40d3041238401dfcb7f4a7fd7. Latest consensus state: current_term: 1 leader_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } } I0506 07:57:26.189365 10168 sys_catalog.cc:338] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:26.193322 10147 sys_catalog.cc:335] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } } I0506 07:57:26.193895 10147 sys_catalog.cc:338] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:26.195354 10147 sys_catalog.cc:422] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:26.197942 10169 sys_catalog.cc:335] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9cc35a40d3041238401dfcb7f4a7fd7" member_type: VOTER } } I0506 07:57:26.220973 10169 sys_catalog.cc:338] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:26.235765 10174 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:26.258729 10173 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:26.265287 10044 master_main.cc:79] Master server successfully started. I0506 07:57:26.283144 10173 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:57:26.395503 10098 tablet.cc:1678] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:26.453140 10173 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:26.453701 10173 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:26.479434 10173 catalog_manager.cc:3992] T 00000000000000000000000000000000 P c9cc35a40d3041238401dfcb7f4a7fd7: Generated new TSK 0 I0506 07:57:26.509374 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 10044 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (4971 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): setting up network... May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): skipping unrecognized local address family 17 May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): skipping unrecognized local address family 17 May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): set up 1 sockets May 06 07:57:26 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): commencing operation krb5kdc: starting... W0506 07:57:27.012157 785 mini_kdc.cc:117] Time spent starting KDC: real 0.365s user 0.004s sys 0.013s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. W0506 07:57:27.207301 785 mini_kdc.cc:250] Time spent creating user principal test-user: real 0.101s user 0.000s sys 0.005s WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. W0506 07:57:27.331534 785 mini_kdc.cc:250] Time spent creating user principal joe-interloper: real 0.124s user 0.002s sys 0.004s May 06 07:57:27 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:27.449064 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:59399 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:27.786674 10233 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:27.787178 10233 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:27.787536 10233 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:27.802423 10233 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:27.802583 10233 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:27.802724 10233 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:27.802845 10233 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:27.802980 10233 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:27.803122 10233 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:27.828184 10233 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:59399 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:27.836798 10233 master_main.cc:73] Initializing master server... I0506 07:57:27.840145 10233 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 219178us May 06 07:57:27 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:27.846451 10233 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:27.852321 10233 fs_manager.cc:260] Metadata directory not provided I0506 07:57:27.852468 10233 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:27.852725 10233 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:27.852854 10233 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:27.865311 10233 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "32cfb79b11d74246a0ff44532072e416" format_stamp: "Formatted at 2018-05-06 14:57:27 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:27.866200 10233 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "32cfb79b11d74246a0ff44532072e416" format_stamp: "Formatted at 2018-05-06 14:57:27 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:27.879277 10233 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s I0506 07:57:27.880542 10233 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:27.881795 10233 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:27.891114 10233 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s I0506 07:57:27.891244 10233 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/wal uuid: "32cfb79b11d74246a0ff44532072e416" format_stamp: "Formatted at 2018-05-06 14:57:27 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:27.891491 10233 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:28.102051 10233 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:28.107038 10233 master_main.cc:76] Starting Master server... I0506 07:57:28.207129 10233 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:59399 I0506 07:57:28.208030 10233 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:28.208142 10233 webserver.cc:184] Document root disabled I0506 07:57:28.210005 10233 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:36266/ I0506 07:57:28.210764 10233 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:28.214329 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 10233 I0506 07:57:28.214681 10422 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:28.246074 10422 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: Bootstrap starting. I0506 07:57:28.249570 10422 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: No blocks or log segments found. Creating new log. I0506 07:57:28.251667 10422 log.cc:520] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: Log is configured to *not* fsync() on all Append() calls I0506 07:57:28.256212 10422 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: No bootstrap required, opened a new log I0506 07:57:28.260809 10424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:28.217816 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:40070 0506 07:57:28.220729 (+ 2913us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:28.220763 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:28.220818 (+ 55us) server_negotiation.cc:373] Connection header received 0506 07:57:28.233245 (+ 12427us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:28.233289 (+ 44us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:28.233550 (+ 261us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:28.234810 (+ 1260us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:28.235933 (+ 1123us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:28.236608 (+ 675us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:28.237947 (+ 1339us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:28.239370 (+ 1423us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:28.239602 (+ 232us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:28.244784 (+ 5182us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:28.244841 (+ 57us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:28.244861 (+ 20us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:28.244916 (+ 55us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:28.251373 (+ 6457us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:28.254342 (+ 2969us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:28.254360 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:28.254370 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:28.255069 (+ 699us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:28.256110 (+ 1041us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:28.256134 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:28.256144 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:28.256564 (+ 420us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:28.257205 (+ 641us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:28.258100 (+ 895us) server_negotiation.cc:264] Negotiation successful 0506 07:57:28.258479 (+ 379us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2599,"thread_start_us":2132,"threads_started":1} I0506 07:57:28.263875 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:28.297899 10424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:28.275666 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:59844 0506 07:57:28.275937 (+ 271us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:28.275958 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:28.277460 (+ 1502us) server_negotiation.cc:373] Connection header received 0506 07:57:28.277756 (+ 296us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:28.277781 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:28.277892 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:28.278018 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:28.280082 (+ 2064us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:28.280428 (+ 346us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:28.282095 (+ 1667us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:28.282943 (+ 848us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:28.283058 (+ 115us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:28.286084 (+ 3026us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:28.286147 (+ 63us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:28.286171 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:28.286239 (+ 68us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:28.290508 (+ 4269us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:28.292526 (+ 2018us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:28.292541 (+ 15us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:28.292550 (+ 9us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:28.293092 (+ 542us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:28.294194 (+ 1102us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:28.294219 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:28.294237 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:28.294716 (+ 479us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:28.295207 (+ 491us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:28.296171 (+ 964us) server_negotiation.cc:264] Negotiation successful 0506 07:57:28.296993 (+ 822us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":95} I0506 07:57:28.269093 10422 raft_consensus.cc:303] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } I0506 07:57:28.346643 10422 raft_consensus.cc:329] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:28.346806 10422 raft_consensus.cc:604] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32cfb79b11d74246a0ff44532072e416, State: Initialized, Role: FOLLOWER I0506 07:57:28.347339 10422 consensus_queue.cc:227] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } I0506 07:57:28.347734 10422 raft_consensus.cc:343] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:28.347944 10422 raft_consensus.cc:435] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:28.348184 10422 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: Snoozing failure detection for 1.641s (starting election) I0506 07:57:28.348378 10422 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:28.351083 10422 raft_consensus.cc:457] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } I0506 07:57:28.351534 10422 leader_election.cc:257] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:28.355098 10431 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: Snoozing failure detection for 1.911s (election complete) I0506 07:57:28.355371 10431 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:28.359171 10431 raft_consensus.cc:571] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [term 1 LEADER]: Becoming Leader. State: Replica: 32cfb79b11d74246a0ff44532072e416, State: Running, Role: LEADER I0506 07:57:28.360160 10431 consensus_queue.cc:204] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } I0506 07:57:28.364926 10431 sys_catalog.cc:335] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32cfb79b11d74246a0ff44532072e416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } } I0506 07:57:28.365945 10431 sys_catalog.cc:338] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: This master's current role is: LEADER I0506 07:57:28.366243 10433 sys_catalog.cc:335] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32cfb79b11d74246a0ff44532072e416. Latest consensus state: current_term: 1 leader_uuid: "32cfb79b11d74246a0ff44532072e416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } } I0506 07:57:28.367027 10433 sys_catalog.cc:338] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: This master's current role is: LEADER I0506 07:57:28.370379 10422 sys_catalog.cc:335] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "32cfb79b11d74246a0ff44532072e416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32cfb79b11d74246a0ff44532072e416" member_type: VOTER } } I0506 07:57:28.370824 10422 sys_catalog.cc:338] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: This master's current role is: LEADER I0506 07:57:28.372225 10422 sys_catalog.cc:422] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:28.377511 10233 master_main.cc:79] Master server successfully started. W0506 07:57:28.382089 10436 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:28.385323 10436 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:28.385628 10435 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:28.388654 10435 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:28.446358 10435 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:28.447885 10435 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:28.460821 10435 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 32cfb79b11d74246a0ff44532072e416: Generated new TSK 0 I0506 07:57:28.487815 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 10233 W0506 07:57:28.509063 10188 connection.cc:512] client connection to 127.0.196.126:59399 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.126:59399): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1918 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): setting up network... May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): skipping unrecognized local address family 17 May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): skipping unrecognized local address family 17 May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): set up 1 sockets May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): commencing operation krb5kdc: starting... WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:28.721055 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:41928 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:28.887426 10451 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:28.887750 10451 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:28.887979 10451 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:28.896952 10451 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:28.897089 10451 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:28.897187 10451 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:28.897281 10451 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:28.897390 10451 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:28.897507 10451 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:28.916864 10451 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:41928 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:28.925205 10451 master_main.cc:73] Initializing master server... I0506 07:57:28.927646 10451 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 219678us May 06 07:57:28 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:28.933782 10451 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:28.939534 10451 fs_manager.cc:260] Metadata directory not provided I0506 07:57:28.939689 10451 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:28.939967 10451 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:28.940136 10451 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:28.949815 10451 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" format_stamp: "Formatted at 2018-05-06 14:57:28 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:28.950561 10451 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" format_stamp: "Formatted at 2018-05-06 14:57:28 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:28.962806 10451 fs_manager.cc:495] Time spent creating directory manager: real 0.012s user 0.008s sys 0.003s I0506 07:57:28.964172 10451 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:28.965580 10451 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:28.973819 10451 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0506 07:57:28.973942 10451 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/wal uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" format_stamp: "Formatted at 2018-05-06 14:57:28 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:28.974177 10451 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:29.146595 10451 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:29.149793 10451 master_main.cc:76] Starting Master server... I0506 07:57:29.295135 10451 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:41928 I0506 07:57:29.296495 10451 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:29.296695 10451 webserver.cc:184] Document root disabled I0506 07:57:29.299186 10451 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:40998/ I0506 07:57:29.300196 10451 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:29.303141 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 10451 I0506 07:57:29.306072 10581 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:29.358685 10581 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: Bootstrap starting. I0506 07:57:29.362632 10581 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: No blocks or log segments found. Creating new log. I0506 07:57:29.366154 10581 log.cc:520] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: Log is configured to *not* fsync() on all Append() calls I0506 07:57:29.376346 10583 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:29.306474 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:38676 0506 07:57:29.320851 (+ 14377us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:29.320879 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:29.320945 (+ 66us) server_negotiation.cc:373] Connection header received 0506 07:57:29.335511 (+ 14566us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:29.335561 (+ 50us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:29.335900 (+ 339us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:29.336259 (+ 359us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:29.339203 (+ 2944us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:29.340004 (+ 801us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:29.347742 (+ 7738us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:29.350319 (+ 2577us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:29.350538 (+ 219us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:29.355799 (+ 5261us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:29.355848 (+ 49us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:29.355883 (+ 35us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:29.355984 (+ 101us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:29.367064 (+ 11080us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:29.368968 (+ 1904us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:29.368995 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:29.369013 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:29.369717 (+ 704us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:29.370222 (+ 505us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:29.370246 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:29.370262 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:29.371012 (+ 750us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:29.371519 (+ 507us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:29.371995 (+ 476us) server_negotiation.cc:264] Negotiation successful 0506 07:57:29.373518 (+ 1523us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14050,"thread_start_us":13358,"threads_started":1} I0506 07:57:29.379629 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:29.447053 10583 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:29.411368 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:41068 0506 07:57:29.411691 (+ 323us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:29.411715 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:29.419274 (+ 7559us) server_negotiation.cc:373] Connection header received 0506 07:57:29.419548 (+ 274us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:29.419575 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:29.419729 (+ 154us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:29.419861 (+ 132us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:29.422254 (+ 2393us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:29.422568 (+ 314us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:29.424283 (+ 1715us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:29.425781 (+ 1498us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:29.425963 (+ 182us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:29.429890 (+ 3927us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:29.429957 (+ 67us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:29.429982 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:29.430058 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:29.436584 (+ 6526us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:29.439357 (+ 2773us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:29.439383 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:29.439401 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:29.440068 (+ 667us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:29.441614 (+ 1546us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:29.441641 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:29.441685 (+ 44us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:29.442153 (+ 468us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:29.442630 (+ 477us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:29.445175 (+ 2545us) server_negotiation.cc:264] Negotiation successful 0506 07:57:29.445515 (+ 340us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} I0506 07:57:29.376986 10581 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: No bootstrap required, opened a new log I0506 07:57:29.482532 10581 raft_consensus.cc:303] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } I0506 07:57:29.483116 10581 raft_consensus.cc:329] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:29.483343 10581 raft_consensus.cc:604] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e12cd79ada24fd8bb8ae976ff5484f7, State: Initialized, Role: FOLLOWER I0506 07:57:29.484002 10581 consensus_queue.cc:227] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } I0506 07:57:29.484534 10581 raft_consensus.cc:343] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:29.484817 10581 raft_consensus.cc:435] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:29.485074 10581 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: Snoozing failure detection for 1.667s (starting election) I0506 07:57:29.485327 10581 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:29.489280 10581 raft_consensus.cc:457] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } I0506 07:57:29.489936 10581 leader_election.cc:257] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:29.493000 10594 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: Snoozing failure detection for 2.015s (election complete) I0506 07:57:29.493324 10594 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:29.502773 10594 raft_consensus.cc:571] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e12cd79ada24fd8bb8ae976ff5484f7, State: Running, Role: LEADER I0506 07:57:29.503311 10594 consensus_queue.cc:204] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } I0506 07:57:29.515699 10596 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e12cd79ada24fd8bb8ae976ff5484f7. Latest consensus state: current_term: 1 leader_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } } I0506 07:57:29.515955 10595 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } } I0506 07:57:29.516319 10596 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:29.516494 10595 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:29.516968 10581 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e12cd79ada24fd8bb8ae976ff5484f7" member_type: VOTER } } I0506 07:57:29.517356 10581 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: This master's current role is: LEADER I0506 07:57:29.518564 10581 sys_catalog.cc:422] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:29.529778 10451 master_main.cc:79] Master server successfully started. W0506 07:57:29.537845 10604 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:29.538166 10604 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:29.538533 10603 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:29.542618 10603 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:29.656786 10603 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:29.659232 10603 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:29.680392 10603 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 9e12cd79ada24fd8bb8ae976ff5484f7: Generated new TSK 0 I0506 07:57:29.689901 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 10451 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1219 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): setting up network... May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): skipping unrecognized local address family 17 May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): skipping unrecognized local address family 17 May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): set up 1 sockets May 06 07:57:29 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): commencing operation krb5kdc: starting... W0506 07:57:29.900168 785 mini_kdc.cc:117] Time spent starting KDC: real 0.159s user 0.000s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:30 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:30.091043 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:48104 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:30.359097 10691 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:30.359689 10691 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:30.360091 10691 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:30.377346 10691 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:30.377602 10691 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:30.377815 10691 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:30.377998 10691 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:30.378222 10691 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:30.378470 10691 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:30.409668 10691 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:48104 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:30.423954 10691 master_main.cc:73] Initializing master server... I0506 07:57:30.427357 10691 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 220678us May 06 07:57:30 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:30.434283 10691 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:30.449700 10691 fs_manager.cc:260] Metadata directory not provided I0506 07:57:30.449936 10691 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:30.450282 10691 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:30.450515 10691 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:30.466403 10691 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "e1765e2e0e504afa9af0ace35c7914ae" format_stamp: "Formatted at 2018-05-06 14:57:30 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:30.467635 10691 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "e1765e2e0e504afa9af0ace35c7914ae" format_stamp: "Formatted at 2018-05-06 14:57:30 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:30.495137 10691 fs_manager.cc:495] Time spent creating directory manager: real 0.027s user 0.016s sys 0.007s I0506 07:57:30.497035 10691 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:30.498957 10691 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:30.516878 10691 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s I0506 07:57:30.517102 10691 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/wal uuid: "e1765e2e0e504afa9af0ace35c7914ae" format_stamp: "Formatted at 2018-05-06 14:57:30 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:30.517508 10691 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:30.879701 10691 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:30.888379 10691 master_main.cc:76] Starting Master server... I0506 07:57:31.208209 10691 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:48104 I0506 07:57:31.209185 10691 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:31.209298 10691 webserver.cc:184] Document root disabled I0506 07:57:31.211560 10691 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:33247/ I0506 07:57:31.212319 10691 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:31.213379 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 10691 I0506 07:57:31.218828 10910 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:31.255982 10910 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: Bootstrap starting. I0506 07:57:31.259953 10910 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: No blocks or log segments found. Creating new log. I0506 07:57:31.263252 10910 log.cc:520] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: Log is configured to *not* fsync() on all Append() calls I0506 07:57:31.266692 10912 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:31.216132 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56697 0506 07:57:31.243868 (+ 27736us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:31.243899 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:31.243956 (+ 57us) server_negotiation.cc:373] Connection header received 0506 07:57:31.248344 (+ 4388us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:31.248392 (+ 48us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:31.248694 (+ 302us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:31.249044 (+ 350us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:31.251454 (+ 2410us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:31.252338 (+ 884us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:31.255080 (+ 2742us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:31.257314 (+ 2234us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:31.259445 (+ 2131us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:31.260242 (+ 797us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:31.260318 (+ 76us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:31.260350 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:31.261163 (+ 813us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:31.262146 (+ 983us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:31.262273 (+ 127us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:31.263894 (+ 1621us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:31.264396 (+ 502us) server_negotiation.cc:264] Negotiation successful 0506 07:57:31.264526 (+ 130us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":27397,"thread_start_us":27014,"threads_started":1} I0506 07:57:31.272318 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:31.273833 10910 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: No bootstrap required, opened a new log I0506 07:57:31.285665 10910 raft_consensus.cc:303] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } I0506 07:57:31.286027 10910 raft_consensus.cc:329] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:31.286244 10910 raft_consensus.cc:604] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1765e2e0e504afa9af0ace35c7914ae, State: Initialized, Role: FOLLOWER I0506 07:57:31.286878 10910 consensus_queue.cc:227] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } I0506 07:57:31.287292 10910 raft_consensus.cc:343] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:31.287472 10910 raft_consensus.cc:435] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:31.287653 10910 raft_consensus.cc:2704] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: Snoozing failure detection for 1.939s (starting election) I0506 07:57:31.287825 10910 raft_consensus.cc:2756] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:31.291033 10910 raft_consensus.cc:457] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } I0506 07:57:31.291676 10910 leader_election.cc:257] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:31.296751 10926 raft_consensus.cc:2704] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: Snoozing failure detection for 1.839s (election complete) I0506 07:57:31.299425 10926 raft_consensus.cc:2511] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:31.304054 10926 raft_consensus.cc:571] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [term 1 LEADER]: Becoming Leader. State: Replica: e1765e2e0e504afa9af0ace35c7914ae, State: Running, Role: LEADER I0506 07:57:31.304543 10926 consensus_queue.cc:204] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } I0506 07:57:31.308054 10912 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:31.294682 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:49553 0506 07:57:31.294955 (+ 273us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:31.294977 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:31.296812 (+ 1835us) server_negotiation.cc:373] Connection header received 0506 07:57:31.297072 (+ 260us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:31.297098 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:31.297215 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:31.297393 (+ 178us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:31.298520 (+ 1127us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:31.299288 (+ 768us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:31.302092 (+ 2804us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:31.303542 (+ 1450us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:31.303736 (+ 194us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:31.305236 (+ 1500us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:31.305300 (+ 64us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:31.305321 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:31.305363 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:31.305435 (+ 72us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:31.305501 (+ 66us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:31.305846 (+ 345us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:31.306564 (+ 718us) server_negotiation.cc:264] Negotiation successful 0506 07:57:31.306656 (+ 92us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":84} I0506 07:57:31.322479 10910 sys_catalog.cc:335] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e1765e2e0e504afa9af0ace35c7914ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } } I0506 07:57:31.426704 10910 sys_catalog.cc:338] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: This master's current role is: LEADER I0506 07:57:31.322852 10928 sys_catalog.cc:335] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1765e2e0e504afa9af0ace35c7914ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } } I0506 07:57:31.449566 10928 sys_catalog.cc:338] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: This master's current role is: LEADER I0506 07:57:31.323212 10929 sys_catalog.cc:335] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1765e2e0e504afa9af0ace35c7914ae. Latest consensus state: current_term: 1 leader_uuid: "e1765e2e0e504afa9af0ace35c7914ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1765e2e0e504afa9af0ace35c7914ae" member_type: VOTER } } I0506 07:57:31.453938 10929 sys_catalog.cc:338] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: This master's current role is: LEADER I0506 07:57:31.456197 10910 sys_catalog.cc:422] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:31.456585 10950 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:31.459264 10953 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:31.461067 10691 master_main.cc:79] Master server successfully started. I0506 07:57:31.468441 10950 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:31.564651 10950 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:31.567271 10950 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:31.591822 10950 catalog_manager.cc:3992] T 00000000000000000000000000000000 P e1765e2e0e504afa9af0ace35c7914ae: Generated new TSK 0 I0506 07:57:31.621094 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 10691 W0506 07:57:31.659672 10620 connection.cc:512] client connection to 127.0.196.126:48104 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.126:48104): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1933 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): setting up network... May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): skipping unrecognized local address family 17 May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): skipping unrecognized local address family 17 May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): set up 1 sockets May 06 07:57:31 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): commencing operation krb5kdc: starting... W0506 07:57:31.980116 785 mini_kdc.cc:117] Time spent starting KDC: real 0.301s user 0.003s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:32.240411 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51804 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:32.525867 11000 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:32.526363 11000 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:32.526808 11000 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:32.544484 11000 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:32.544754 11000 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:32.544951 11000 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:32.545143 11000 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:32.545364 11000 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:32.545600 11000 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:32.590858 11000 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51804 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:32.606050 11000 master_main.cc:73] Initializing master server... I0506 07:57:32.609249 11000 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 221678us May 06 07:57:32 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[10985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:32.615833 11000 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:32.629231 11000 fs_manager.cc:260] Metadata directory not provided I0506 07:57:32.629467 11000 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:32.629848 11000 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:32.630084 11000 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:32.646072 11000 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "684134e033cd460ca59a1e7b15439b4f" format_stamp: "Formatted at 2018-05-06 14:57:32 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:32.647215 11000 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "684134e033cd460ca59a1e7b15439b4f" format_stamp: "Formatted at 2018-05-06 14:57:32 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:32.685250 11000 fs_manager.cc:495] Time spent creating directory manager: real 0.037s user 0.012s sys 0.012s I0506 07:57:32.687091 11000 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:32.688529 11000 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:32.709815 11000 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.002s sys 0.005s I0506 07:57:32.710041 11000 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/wal uuid: "684134e033cd460ca59a1e7b15439b4f" format_stamp: "Formatted at 2018-05-06 14:57:32 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:32.710417 11000 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:32.935173 11000 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:32.942622 11000 master_main.cc:76] Starting Master server... I0506 07:57:33.164515 11000 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:51804 I0506 07:57:33.165796 11000 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:33.165938 11000 webserver.cc:184] Document root disabled I0506 07:57:33.171737 11000 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:42293/ I0506 07:57:33.172737 11000 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:33.174587 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 11000 I0506 07:57:33.192149 11136 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:33.235682 11138 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:33.176985 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:58577 0506 07:57:33.180938 (+ 3953us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:33.180983 (+ 45us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:33.181269 (+ 286us) server_negotiation.cc:373] Connection header received 0506 07:57:33.218358 (+ 37089us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:33.218407 (+ 49us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:33.218723 (+ 316us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:33.220329 (+ 1606us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:33.220764 (+ 435us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:33.221642 (+ 878us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:33.225911 (+ 4269us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:33.228143 (+ 2232us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:33.229579 (+ 1436us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:33.229973 (+ 394us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:33.230007 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:33.230037 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:33.230634 (+ 597us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:33.230738 (+ 104us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:33.230815 (+ 77us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:33.232420 (+ 1605us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:33.232545 (+ 125us) server_negotiation.cc:264] Negotiation successful 0506 07:57:33.232815 (+ 270us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1739,"thread_start_us":3692,"threads_started":1} I0506 07:57:33.238127 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:33.238209 11136 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: Bootstrap starting. I0506 07:57:33.241147 11136 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: No blocks or log segments found. Creating new log. I0506 07:57:33.243479 11136 log.cc:520] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: Log is configured to *not* fsync() on all Append() calls I0506 07:57:33.252992 11136 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: No bootstrap required, opened a new log I0506 07:57:33.264672 11136 raft_consensus.cc:303] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } I0506 07:57:33.265019 11136 raft_consensus.cc:329] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:33.265177 11136 raft_consensus.cc:604] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684134e033cd460ca59a1e7b15439b4f, State: Initialized, Role: FOLLOWER I0506 07:57:33.266957 11136 consensus_queue.cc:227] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } I0506 07:57:33.267438 11136 raft_consensus.cc:343] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:33.267715 11136 raft_consensus.cc:435] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:33.267961 11136 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: Snoozing failure detection for 1.568s (starting election) I0506 07:57:33.268198 11136 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:33.272749 11136 raft_consensus.cc:457] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } I0506 07:57:33.273370 11136 leader_election.cc:257] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:33.275285 11145 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: Snoozing failure detection for 1.985s (election complete) I0506 07:57:33.275615 11145 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:33.281618 11145 raft_consensus.cc:571] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [term 1 LEADER]: Becoming Leader. State: Replica: 684134e033cd460ca59a1e7b15439b4f, State: Running, Role: LEADER I0506 07:57:33.281972 11145 consensus_queue.cc:204] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } I0506 07:57:33.291355 11138 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:33.266437 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:54765 0506 07:57:33.266752 (+ 315us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:33.266773 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:33.273800 (+ 7027us) server_negotiation.cc:373] Connection header received 0506 07:57:33.274057 (+ 257us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:33.274081 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:33.274183 (+ 102us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:33.274296 (+ 113us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:33.282708 (+ 8412us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:33.283220 (+ 512us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:33.285145 (+ 1925us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:33.286570 (+ 1425us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:33.286760 (+ 190us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:33.288282 (+ 1522us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:33.288355 (+ 73us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:33.288383 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:33.288439 (+ 56us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:33.288519 (+ 80us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:33.288621 (+ 102us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:33.289006 (+ 385us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:33.289799 (+ 793us) server_negotiation.cc:264] Negotiation successful 0506 07:57:33.289885 (+ 86us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":106} I0506 07:57:33.296334 11136 sys_catalog.cc:335] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "684134e033cd460ca59a1e7b15439b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } } I0506 07:57:33.361634 11136 sys_catalog.cc:338] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: This master's current role is: LEADER I0506 07:57:33.296752 11147 sys_catalog.cc:335] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 684134e033cd460ca59a1e7b15439b4f. Latest consensus state: current_term: 1 leader_uuid: "684134e033cd460ca59a1e7b15439b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } } I0506 07:57:33.377825 11147 sys_catalog.cc:338] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: This master's current role is: LEADER I0506 07:57:33.296897 11146 sys_catalog.cc:335] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "684134e033cd460ca59a1e7b15439b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "684134e033cd460ca59a1e7b15439b4f" member_type: VOTER } } I0506 07:57:33.382081 11146 sys_catalog.cc:338] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: This master's current role is: LEADER I0506 07:57:33.388159 11136 sys_catalog.cc:422] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:33.388427 11149 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:33.395095 11151 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:33.397307 11000 master_main.cc:79] Master server successfully started. I0506 07:57:33.402324 11149 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:33.460098 11149 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:33.462661 11149 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:33.481726 11149 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 684134e033cd460ca59a1e7b15439b4f: Generated new TSK 0 I0506 07:57:33.502002 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 11000 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1872 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): setting up network... May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): skipping unrecognized local address family 17 May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): skipping unrecognized local address family 17 May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): set up 1 sockets May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): commencing operation krb5kdc: starting... W0506 07:57:33.756578 785 mini_kdc.cc:117] Time spent starting KDC: real 0.204s user 0.002s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:33 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:33.936215 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:41685 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:34.175338 11242 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:34.175781 11242 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:34.176174 11242 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:34.194420 11242 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:34.194694 11242 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:34.194898 11242 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:34.195102 11242 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:34.195324 11242 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:34.195560 11242 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:34.242146 11242 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:41685 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:34.257035 11242 master_main.cc:73] Initializing master server... I0506 07:57:34.259459 11242 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 222678us May 06 07:57:34 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:34.262857 11242 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:34.274659 11242 fs_manager.cc:260] Metadata directory not provided I0506 07:57:34.274889 11242 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:34.275228 11242 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:34.275458 11242 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:34.291712 11242 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "9243f7b5f809452e88fe7616e16ff9d1" format_stamp: "Formatted at 2018-05-06 14:57:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:34.292755 11242 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "9243f7b5f809452e88fe7616e16ff9d1" format_stamp: "Formatted at 2018-05-06 14:57:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:34.322155 11242 fs_manager.cc:495] Time spent creating directory manager: real 0.022s user 0.013s sys 0.005s I0506 07:57:34.324002 11242 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:34.324664 11242 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:34.342350 11242 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.005s I0506 07:57:34.342671 11242 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/wal uuid: "9243f7b5f809452e88fe7616e16ff9d1" format_stamp: "Formatted at 2018-05-06 14:57:34 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:34.343029 11242 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:34.545437 11242 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:34.552641 11242 master_main.cc:76] Starting Master server... I0506 07:57:34.713507 11242 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:41685 I0506 07:57:34.715032 11242 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:34.715214 11242 webserver.cc:184] Document root disabled I0506 07:57:34.716987 11242 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:53085/ I0506 07:57:34.718051 11242 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:34.722815 11379 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:34.726935 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 11242 I0506 07:57:34.758462 11385 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:34.728779 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:49446 0506 07:57:34.739559 (+ 10780us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:34.739602 (+ 43us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:34.745445 (+ 5843us) server_negotiation.cc:373] Connection header received 0506 07:57:34.750825 (+ 5380us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:34.750880 (+ 55us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:34.751149 (+ 269us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:34.751550 (+ 401us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:34.753150 (+ 1600us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:34.753182 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:34.753210 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:34.753256 (+ 46us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:34.753343 (+ 87us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:34.753419 (+ 76us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:34.753538 (+ 119us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:34.756807 (+ 3269us) server_negotiation.cc:264] Negotiation successful 0506 07:57:34.756940 (+ 133us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10434,"thread_start_us":10637,"threads_started":1} I0506 07:57:34.760768 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:34.770352 11379 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Bootstrap starting. I0506 07:57:34.773844 11379 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: No blocks or log segments found. Creating new log. I0506 07:57:34.785384 11379 log.cc:520] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Log is configured to *not* fsync() on all Append() calls I0506 07:57:34.802245 11379 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: No bootstrap required, opened a new log I0506 07:57:34.804675 11385 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:34.798347 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:52357 0506 07:57:34.798824 (+ 477us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:34.798849 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:34.801386 (+ 2537us) server_negotiation.cc:373] Connection header received 0506 07:57:34.801679 (+ 293us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:34.801708 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:34.801813 (+ 105us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:34.801952 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:34.802592 (+ 640us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:34.802619 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:34.802672 (+ 53us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:34.802721 (+ 49us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:34.802814 (+ 93us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:34.802902 (+ 88us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:34.803033 (+ 131us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:34.803359 (+ 326us) server_negotiation.cc:264] Negotiation successful 0506 07:57:34.803449 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":201} I0506 07:57:34.849715 11379 raft_consensus.cc:303] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } I0506 07:57:34.955657 11379 raft_consensus.cc:329] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:34.955883 11379 raft_consensus.cc:604] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9243f7b5f809452e88fe7616e16ff9d1, State: Initialized, Role: FOLLOWER I0506 07:57:34.956565 11379 consensus_queue.cc:227] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } I0506 07:57:34.957108 11379 raft_consensus.cc:343] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:34.957375 11379 raft_consensus.cc:435] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:34.957664 11379 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Snoozing failure detection for 1.925s (starting election) I0506 07:57:34.957918 11379 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:34.962576 11379 raft_consensus.cc:457] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } I0506 07:57:34.963289 11379 leader_election.cc:257] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:34.969967 11433 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Snoozing failure detection for 1.973s (election complete) I0506 07:57:34.971145 11433 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:34.976166 11433 raft_consensus.cc:571] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [term 1 LEADER]: Becoming Leader. State: Replica: 9243f7b5f809452e88fe7616e16ff9d1, State: Running, Role: LEADER I0506 07:57:34.976675 11433 consensus_queue.cc:204] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } I0506 07:57:34.986639 11379 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "9243f7b5f809452e88fe7616e16ff9d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } } I0506 07:57:34.987080 11379 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: This master's current role is: LEADER I0506 07:57:34.988538 11436 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9243f7b5f809452e88fe7616e16ff9d1. Latest consensus state: current_term: 1 leader_uuid: "9243f7b5f809452e88fe7616e16ff9d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } } I0506 07:57:34.989069 11436 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: This master's current role is: LEADER I0506 07:57:34.989552 11434 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9243f7b5f809452e88fe7616e16ff9d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9243f7b5f809452e88fe7616e16ff9d1" member_type: VOTER } } I0506 07:57:34.990005 11434 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: This master's current role is: LEADER I0506 07:57:34.993742 11379 sys_catalog.cc:422] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:34.997315 11242 master_main.cc:79] Master server successfully started. W0506 07:57:35.009096 11440 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:35.009325 11440 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:35.009567 11439 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:35.013844 11439 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:57:35.065310 11325 tablet.cc:1678] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:35.196494 11439 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:35.197031 11439 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:35.221572 11439 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 9243f7b5f809452e88fe7616e16ff9d1: Generated new TSK 0 I0506 07:57:35.262042 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 11242 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1765 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): setting up network... May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): skipping unrecognized local address family 17 May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): skipping unrecognized local address family 17 May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): set up 1 sockets May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): commencing operation krb5kdc: starting... W0506 07:57:35.480254 785 mini_kdc.cc:117] Time spent starting KDC: real 0.163s user 0.002s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:35 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:35.703476 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:42420 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:35.911985 11539 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:35.915269 11539 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:35.920312 11539 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:35.949204 11539 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:35.949451 11539 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:35.949708 11539 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:35.949909 11539 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:35.950135 11539 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:35.950369 11539 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:35.987807 11539 master_main.cc:66] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:42420 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:36.059164 11539 master_main.cc:73] Initializing master server... I0506 07:57:36.061616 11539 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 223678us May 06 07:57:36 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:36.065014 11539 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:36.096760 11539 fs_manager.cc:260] Metadata directory not provided I0506 07:57:36.097534 11539 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:36.097929 11539 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:36.098632 11539 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:36.115586 11539 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "5f5b2ca092e54ba098ad5797bb96ff94" format_stamp: "Formatted at 2018-05-06 14:57:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:36.117415 11539 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "5f5b2ca092e54ba098ad5797bb96ff94" format_stamp: "Formatted at 2018-05-06 14:57:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:36.157778 11539 fs_manager.cc:495] Time spent creating directory manager: real 0.040s user 0.013s sys 0.002s I0506 07:57:36.163957 11539 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:36.164652 11539 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:36.201092 11539 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.001s I0506 07:57:36.201328 11539 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/wal uuid: "5f5b2ca092e54ba098ad5797bb96ff94" format_stamp: "Formatted at 2018-05-06 14:57:36 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:36.201784 11539 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:36.507074 11539 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:36.511180 11539 master_main.cc:76] Starting Master server... I0506 07:57:36.878841 11539 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:42420 I0506 07:57:36.881144 11539 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:36.881335 11539 webserver.cc:184] Document root disabled I0506 07:57:36.887382 11539 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:40671/ I0506 07:57:36.888203 11539 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:36.888830 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 11539 I0506 07:57:36.907779 11674 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:36.937999 11676 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:36.891524 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:37017 0506 07:57:36.904843 (+ 13319us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:36.904871 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:36.904925 (+ 54us) server_negotiation.cc:373] Connection header received 0506 07:57:36.929465 (+ 24540us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:36.929523 (+ 58us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:36.930245 (+ 722us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:36.932755 (+ 2510us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:36.933817 (+ 1062us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:36.933852 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:36.933883 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:36.933934 (+ 51us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:36.934055 (+ 121us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:36.934143 (+ 88us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:36.935177 (+ 1034us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:36.935481 (+ 304us) server_negotiation.cc:264] Negotiation successful 0506 07:57:36.935675 (+ 194us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12925,"thread_start_us":12331,"threads_started":1} I0506 07:57:36.940660 785 external_mini_cluster.cc:482] 0 TS(s) registered with all masters I0506 07:57:36.953716 11674 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Bootstrap starting. I0506 07:57:36.961352 11674 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: No blocks or log segments found. Creating new log. I0506 07:57:36.963096 11674 log.cc:520] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Log is configured to *not* fsync() on all Append() calls I0506 07:57:36.973027 11674 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: No bootstrap required, opened a new log I0506 07:57:36.994191 11676 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:36.985177 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:55755 0506 07:57:36.985713 (+ 536us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:36.985736 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:36.989427 (+ 3691us) server_negotiation.cc:373] Connection header received 0506 07:57:36.990693 (+ 1266us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:36.990719 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:36.990819 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:36.990945 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:36.992045 (+ 1100us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:36.992073 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:36.992095 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0506 07:57:36.992140 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:36.992211 (+ 71us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0506 07:57:36.992278 (+ 67us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:36.992405 (+ 127us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:36.992904 (+ 499us) server_negotiation.cc:264] Negotiation successful 0506 07:57:36.992993 (+ 89us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":265} I0506 07:57:36.997020 11674 raft_consensus.cc:303] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } I0506 07:57:37.122982 11674 raft_consensus.cc:329] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0506 07:57:37.123205 11674 raft_consensus.cc:604] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5b2ca092e54ba098ad5797bb96ff94, State: Initialized, Role: FOLLOWER I0506 07:57:37.123898 11674 consensus_queue.cc:227] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } I0506 07:57:37.124454 11674 raft_consensus.cc:343] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0506 07:57:37.124739 11674 raft_consensus.cc:435] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0506 07:57:37.124990 11674 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Snoozing failure detection for 1.651s (starting election) I0506 07:57:37.125226 11674 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:37.132907 11674 raft_consensus.cc:457] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } I0506 07:57:37.133679 11674 leader_election.cc:257] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:37.139660 11696 raft_consensus.cc:2704] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Snoozing failure detection for 2.072s (election complete) I0506 07:57:37.140010 11696 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:37.145382 11696 raft_consensus.cc:571] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [term 1 LEADER]: Becoming Leader. State: Replica: 5f5b2ca092e54ba098ad5797bb96ff94, State: Running, Role: LEADER I0506 07:57:37.146006 11696 consensus_queue.cc:204] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } I0506 07:57:37.155637 11674 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } } I0506 07:57:37.157168 11674 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: This master's current role is: LEADER I0506 07:57:37.156656 11698 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } } I0506 07:57:37.158087 11698 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: This master's current role is: LEADER I0506 07:57:37.157800 11697 sys_catalog.cc:335] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f5b2ca092e54ba098ad5797bb96ff94. Latest consensus state: current_term: 1 leader_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f5b2ca092e54ba098ad5797bb96ff94" member_type: VOTER } } I0506 07:57:37.158440 11697 sys_catalog.cc:338] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: This master's current role is: LEADER I0506 07:57:37.164621 11674 sys_catalog.cc:422] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:37.165117 11701 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:37.168783 11702 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:37.171504 11539 master_main.cc:79] Master server successfully started. I0506 07:57:37.176719 11701 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... W0506 07:57:37.265602 11598 tablet.cc:1678] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:37.293589 11701 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:37.294214 11701 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:37.316529 11701 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 5f5b2ca092e54ba098ad5797bb96ff94: Generated new TSK 0 I0506 07:57:37.348181 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 11539 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2067 ms) [----------] 12 tests from AuthTokenIssuingTest (25586 ms total) [----------] 2 tests from ConnectToFollowerMasterTest [ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): setting up network... May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): skipping unrecognized local address family 17 May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): skipping unrecognized local address family 17 May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): set up 1 sockets May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): commencing operation krb5kdc: starting... W0506 07:57:37.518810 785 mini_kdc.cc:117] Time spent starting KDC: real 0.137s user 0.002s sys 0.012s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. W0506 07:57:37.687799 785 mini_kdc.cc:250] Time spent creating user principal test-user: real 0.128s user 0.002s sys 0.004s WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:37 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: W0506 07:57:37.879758 785 mini_kdc.cc:276] Time spent kinit for test-admin: real 0.105s user 0.001s sys 0.011s WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:37.964030 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51652 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:38.414494 11750 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:38.416560 11750 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:38.416846 11750 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:38.426311 11750 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:38.459477 11750 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:38.459786 11750 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:38.459990 11750 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:38.460187 11750 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:38.460412 11750 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:38.460649 11750 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:38.567752 11750 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:51652 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:38.595971 11750 master_main.cc:73] Initializing master server... I0506 07:57:38.599961 11750 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 224678us May 06 07:57:38 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:38.609180 11750 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:38.627990 11750 fs_manager.cc:260] Metadata directory not provided I0506 07:57:38.628231 11750 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:38.628593 11750 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:38.628866 11750 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:38.644740 11750 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "2510444e70934eb4b65ad5cbec6bc5c2" format_stamp: "Formatted at 2018-05-06 14:57:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:38.645859 11750 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "2510444e70934eb4b65ad5cbec6bc5c2" format_stamp: "Formatted at 2018-05-06 14:57:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:38.668229 11750 fs_manager.cc:495] Time spent creating directory manager: real 0.022s user 0.013s sys 0.006s I0506 07:57:38.669924 11750 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:38.671314 11750 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:38.679677 11750 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I0506 07:57:38.679813 11750 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/wal uuid: "2510444e70934eb4b65ad5cbec6bc5c2" format_stamp: "Formatted at 2018-05-06 14:57:38 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:38.680044 11750 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:38.902588 11750 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:38.906479 11750 master_main.cc:76] Starting Master server... I0506 07:57:39.084026 11750 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:51652 I0506 07:57:39.085041 11750 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:39.085150 11750 webserver.cc:184] Document root disabled I0506 07:57:39.087306 11750 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:37701/ I0506 07:57:39.088376 11750 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:39.092250 11855 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:39.094511 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 11750 WARNING: no policy specified for kudu/127.0.196.125@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.125@KRBTEST.COM" created. I0506 07:57:39.108232 11855 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. I0506 07:57:39.126989 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.125:38563 --webserver_interface=127.0.196.125 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab --principal=kudu/127.0.196.125 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:39.152040 11862 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.112279 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56409 0506 07:57:39.121962 (+ 9683us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:39.121996 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:39.122550 (+ 554us) server_negotiation.cc:373] Connection header received 0506 07:57:39.126681 (+ 4131us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:39.126710 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:39.126894 (+ 184us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:39.127125 (+ 231us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:39.128813 (+ 1688us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.129434 (+ 621us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.130954 (+ 1520us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.132988 (+ 2034us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.133184 (+ 196us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.137267 (+ 4083us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:39.137305 (+ 38us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:39.137333 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:39.137410 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:39.144032 (+ 6622us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.145656 (+ 1624us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.145690 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.145710 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.146515 (+ 805us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.147116 (+ 601us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.147145 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.147163 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.147799 (+ 636us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:39.148060 (+ 261us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:39.148904 (+ 844us) server_negotiation.cc:264] Negotiation successful 0506 07:57:39.149569 (+ 665us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":9304,"thread_start_us":7596,"threads_started":1} I0506 07:57:39.152132 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.112279 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:39.122359 (+ 10080us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.122428 (+ 69us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:39.126294 (+ 3866us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:39.127223 (+ 929us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:39.127250 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:39.128059 (+ 809us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:39.128234 (+ 175us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.128549 (+ 315us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.128589 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.129643 (+ 1054us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.129687 (+ 44us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.130773 (+ 1086us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.130800 (+ 27us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.133527 (+ 2727us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.133551 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.133934 (+ 383us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.134111 (+ 177us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:39.134129 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:39.137010 (+ 2881us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:39.144198 (+ 7188us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.144228 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.144250 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.145456 (+ 1206us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.146707 (+ 1251us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.146735 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.146754 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.146960 (+ 206us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.147986 (+ 1026us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:39.148016 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:39.148371 (+ 355us) client_negotiation.cc:695] Sending connection context 0506 07:57:39.148674 (+ 303us) client_negotiation.cc:225] Negotiation successful 0506 07:57:39.149302 (+ 628us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9741,"spinlock_wait_cycles":12544,"thread_start_us":8865,"threads_started":1} I0506 07:57:39.155709 11855 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:39.157223 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.156423 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.156685 (+ 262us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.156844 (+ 159us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":68} W0506 07:57:39.157609 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:38563: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) I0506 07:57:39.177021 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } attempt: 1 I0506 07:57:39.178649 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.177792 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.178057 (+ 265us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.178202 (+ 145us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0506 07:57:39.179054 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:38563: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) I0506 07:57:39.227365 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } attempt: 2 I0506 07:57:39.228945 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.228094 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.228357 (+ 263us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.228505 (+ 148us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0506 07:57:39.229243 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:38563: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:39.317281 11865 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:39.318006 11865 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:39.318223 11865 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.125 W0506 07:57:39.318614 11865 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:39.334859 11865 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:39.335077 11865 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:39.335247 11865 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:39.335412 11865 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:39.335609 11865 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:39.335855 11865 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.125 I0506 07:57:39.340571 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } attempt: 3 I0506 07:57:39.341866 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.341211 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.341462 (+ 251us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.341570 (+ 108us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":69} W0506 07:57:39.342188 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:38563: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) I0506 07:57:39.369575 11865 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.125:38563 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab --principal=kudu/127.0.196.125 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/info.pb --webserver_interface=127.0.196.125 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:39.380333 11865 master_main.cc:73] Initializing master server... I0506 07:57:39.382396 11865 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 225178us May 06 07:57:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:39.387725 11865 init.cc:339] Logged in from keytab as kudu/127.0.196.125@KRBTEST.COM (short username kudu) I0506 07:57:39.392673 11865 fs_manager.cc:260] Metadata directory not provided I0506 07:57:39.392984 11865 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:39.393430 11865 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/instance: No such file or directory (error 2) I0506 07:57:39.393676 11865 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:39.403141 11865 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/instance: uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" format_stamp: "Formatted at 2018-05-06 14:57:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:39.403873 11865 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/wal/instance: uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" format_stamp: "Formatted at 2018-05-06 14:57:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:39.416952 11865 fs_manager.cc:495] Time spent creating directory manager: real 0.013s user 0.009s sys 0.002s I0506 07:57:39.418136 11865 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:39.419328 11865 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:39.428714 11865 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0506 07:57:39.428838 11865 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/wal uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" format_stamp: "Formatted at 2018-05-06 14:57:39 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:39.429087 11865 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:39.518528 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } attempt: 4 I0506 07:57:39.520131 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.519278 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.519541 (+ 263us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.519711 (+ 170us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0506 07:57:39.520443 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:38563: Network error: Client connection negotiation failed: client connection to 127.0.196.125:38563: connect: Connection refused (error 111) I0506 07:57:39.668098 11865 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:39.671365 11865 master_main.cc:76] Starting Master server... I0506 07:57:39.807777 11865 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.125:38563 I0506 07:57:39.808756 11865 webserver.cc:173] Starting webserver on 127.0.196.125:0 I0506 07:57:39.808867 11865 webserver.cc:184] Document root disabled I0506 07:57:39.811072 11865 webserver.cc:311] Webserver started. Bound to: http://127.0.196.125:53945/ I0506 07:57:39.811842 11865 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-1/data/info.pb I0506 07:57:39.816345 12106 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:39.818784 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 11865 I0506 07:57:39.823725 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } attempt: 5 WARNING: no policy specified for kudu/127.0.196.124@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.124@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. May 06 07:57:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:39.856802 12106 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:39.858961 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.124:50112 --webserver_interface=127.0.196.124 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab --principal=kudu/127.0.196.124 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:39.890507 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.824470 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.824766 (+ 296us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.824801 (+ 35us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:39.824938 (+ 137us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:39.849230 (+ 24292us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:39.849258 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:39.849790 (+ 532us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:39.849894 (+ 104us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.850021 (+ 127us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.850046 (+ 25us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.850980 (+ 934us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.851028 (+ 48us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.851868 (+ 840us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.851895 (+ 27us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.854383 (+ 2488us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.854410 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.854795 (+ 385us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.854907 (+ 112us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:39.854917 (+ 10us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:39.856966 (+ 2049us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:39.877859 (+ 20893us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.877889 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.877910 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.879229 (+ 1319us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.880215 (+ 986us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.880243 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.880262 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.880429 (+ 167us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.884063 (+ 3634us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:39.884089 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:39.887548 (+ 3459us) client_negotiation.cc:695] Sending connection context 0506 07:57:39.887789 (+ 241us) client_negotiation.cc:225] Negotiation successful 0506 07:57:39.888732 (+ 943us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":3107} I0506 07:57:39.895915 12111 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.860416 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56422 0506 07:57:39.872409 (+ 11993us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:39.872436 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:39.872642 (+ 206us) server_negotiation.cc:373] Connection header received 0506 07:57:39.872919 (+ 277us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:39.872944 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:39.873062 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:39.873196 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:39.876382 (+ 3186us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.876739 (+ 357us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.878375 (+ 1636us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.879297 (+ 922us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.879470 (+ 173us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.883536 (+ 4066us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:39.883566 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:39.883590 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:39.883666 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:39.888519 (+ 4853us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.890213 (+ 1694us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.890240 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.890258 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.890919 (+ 661us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.891613 (+ 694us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.891640 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.891660 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.892167 (+ 507us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:39.892344 (+ 177us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:39.893601 (+ 1257us) server_negotiation.cc:264] Negotiation successful 0506 07:57:39.894057 (+ 456us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11782,"thread_start_us":11982,"threads_started":1} I0506 07:57:39.896864 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.860332 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:39.872369 (+ 12037us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.872481 (+ 112us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:39.872725 (+ 244us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:39.874782 (+ 2057us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:39.874811 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:39.875686 (+ 875us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:39.875916 (+ 230us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.876177 (+ 261us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.876220 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.876902 (+ 682us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.876929 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.878176 (+ 1247us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.878217 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.880279 (+ 2062us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.880306 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.880698 (+ 392us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.881230 (+ 532us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:39.881242 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:39.883264 (+ 2022us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:39.888749 (+ 5485us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.888778 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.888797 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.890024 (+ 1227us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.891113 (+ 1089us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.891136 (+ 23us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.891153 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.891424 (+ 271us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.892474 (+ 1050us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:39.892499 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:39.892892 (+ 393us) client_negotiation.cc:695] Sending connection context 0506 07:57:39.893205 (+ 313us) client_negotiation.cc:225] Negotiation successful 0506 07:57:39.895120 (+ 1915us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":11691,"mutex_wait_us":82,"thread_start_us":11090,"threads_started":1} I0506 07:57:39.894486 12108 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.825458 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59114 0506 07:57:39.841919 (+ 16461us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:39.841953 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:39.842026 (+ 73us) server_negotiation.cc:373] Connection header received 0506 07:57:39.848556 (+ 6530us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:39.848604 (+ 48us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:39.848846 (+ 242us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:39.849150 (+ 304us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:39.850216 (+ 1066us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.850807 (+ 591us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.852049 (+ 1242us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.854177 (+ 2128us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.854399 (+ 222us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.859114 (+ 4715us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:39.859148 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:39.859177 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:39.859346 (+ 169us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:39.877641 (+ 18295us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.879422 (+ 1781us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.879458 (+ 36us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.879480 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.880048 (+ 568us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.882264 (+ 2216us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.882297 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.882317 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.883844 (+ 1527us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:39.884446 (+ 602us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:39.891377 (+ 6931us) server_negotiation.cc:264] Negotiation successful 0506 07:57:39.891950 (+ 573us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":817,"server-negotiator.queue_time_us":16137,"thread_start_us":15754,"threads_started":1} I0506 07:57:39.900638 11855 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:39.900758 12106 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:39.903286 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.901811 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.902758 (+ 947us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.902898 (+ 140us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":300} W0506 07:57:39.903829 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) May 06 07:57:39 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:39.924168 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.901765 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:39.904220 (+ 2455us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.904250 (+ 30us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:39.904368 (+ 118us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:39.905211 (+ 843us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:39.905233 (+ 22us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:39.905860 (+ 627us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:39.905991 (+ 131us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.906183 (+ 192us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.906221 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.907099 (+ 878us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.907122 (+ 23us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.908274 (+ 1152us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:39.908312 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.910758 (+ 2446us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.910782 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:39.911155 (+ 373us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.911297 (+ 142us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:39.911314 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:39.913704 (+ 2390us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:39.919128 (+ 5424us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.919144 (+ 16us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.919154 (+ 10us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.919891 (+ 737us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.920536 (+ 645us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:39.920552 (+ 16us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:39.920562 (+ 10us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:39.920675 (+ 113us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:39.921291 (+ 616us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:39.921315 (+ 24us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:39.921514 (+ 199us) client_negotiation.cc:695] Sending connection context 0506 07:57:39.921672 (+ 158us) client_negotiation.cc:225] Negotiation successful 0506 07:57:39.922155 (+ 483us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2291} I0506 07:57:39.924203 12108 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.902088 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59117 0506 07:57:39.904017 (+ 1929us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:39.904038 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:39.904511 (+ 473us) server_negotiation.cc:373] Connection header received 0506 07:57:39.904614 (+ 103us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:39.904671 (+ 57us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:39.904795 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:39.904917 (+ 122us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:39.906394 (+ 1477us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.906907 (+ 513us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.908512 (+ 1605us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:39.910317 (+ 1805us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:39.910465 (+ 148us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:39.913938 (+ 3473us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:39.913963 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:39.913983 (+ 20us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:39.914046 (+ 63us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:39.918994 (+ 4948us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.920018 (+ 1024us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.920035 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.920044 (+ 9us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.920411 (+ 367us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:39.920813 (+ 402us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:39.920828 (+ 15us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:39.920837 (+ 9us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:39.921154 (+ 317us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:39.921308 (+ 154us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:39.921832 (+ 524us) server_negotiation.cc:264] Negotiation successful 0506 07:57:39.922420 (+ 588us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":1755} I0506 07:57:39.925949 12106 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:39.927481 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.926662 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.926904 (+ 242us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.927073 (+ 169us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":66} W0506 07:57:39.927904 12106 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:39.942226 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 1 I0506 07:57:39.943953 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.942971 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.943393 (+ 422us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.943528 (+ 135us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":240} W0506 07:57:39.944424 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:39.947302 12106 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 1 I0506 07:57:39.948628 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.948010 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.948259 (+ 249us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.948357 (+ 98us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":69} W0506 07:57:39.948920 12106 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:39.992801 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 2 I0506 07:57:39.994419 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.993621 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.993942 (+ 321us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.994093 (+ 151us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":99} W0506 07:57:39.994766 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:39.997261 12106 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 2 I0506 07:57:39.998713 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:39.997896 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:39.998156 (+ 260us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:39.998291 (+ 135us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":83} W0506 07:57:39.999055 12106 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:40.089859 12113 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:40.090514 12113 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:40.090724 12113 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.124 W0506 07:57:40.091095 12113 flags.cc:412] Enabled unsafe flag: --never_fsync=true I0506 07:57:40.095124 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 3 I0506 07:57:40.096546 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.095896 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.096164 (+ 268us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.096269 (+ 105us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":76} W0506 07:57:40.096860 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) W0506 07:57:40.108433 12113 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:40.108659 12113 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:40.108814 12113 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:40.108956 12113 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:40.109122 12113 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:40.109333 12113 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.124 I0506 07:57:40.110399 12106 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 3 I0506 07:57:40.111815 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.111158 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.111420 (+ 262us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.111523 (+ 103us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":74} W0506 07:57:40.112197 12106 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:40.131230 12113 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.124:50112 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab --principal=kudu/127.0.196.124 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/info.pb --webserver_interface=127.0.196.124 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:40.139883 12113 master_main.cc:73] Initializing master server... I0506 07:57:40.142421 12113 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 225678us May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:40.147673 12113 init.cc:339] Logged in from keytab as kudu/127.0.196.124@KRBTEST.COM (short username kudu) I0506 07:57:40.152734 12113 fs_manager.cc:260] Metadata directory not provided I0506 07:57:40.152878 12113 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:40.153102 12113 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/instance: No such file or directory (error 2) I0506 07:57:40.153234 12113 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:40.163300 12113 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/instance: uuid: "28567823d1c94558968cb9d87645f6a9" format_stamp: "Formatted at 2018-05-06 14:57:40 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:40.164031 12113 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/wal/instance: uuid: "28567823d1c94558968cb9d87645f6a9" format_stamp: "Formatted at 2018-05-06 14:57:40 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:40.175406 12113 fs_manager.cc:495] Time spent creating directory manager: real 0.011s user 0.007s sys 0.003s I0506 07:57:40.176600 12113 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:40.178045 12113 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:40.186915 12113 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0506 07:57:40.187044 12113 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/wal uuid: "28567823d1c94558968cb9d87645f6a9" format_stamp: "Formatted at 2018-05-06 14:57:40 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:40.187294 12113 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:40.237187 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 4 I0506 07:57:40.238531 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.237859 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.238131 (+ 272us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.238255 (+ 124us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0506 07:57:40.238893 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:40.288556 12106 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 4 I0506 07:57:40.290102 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.289403 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.289708 (+ 305us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.289827 (+ 119us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0506 07:57:40.290498 12106 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:40.399922 12113 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:40.404835 12113 master_main.cc:76] Starting Master server... I0506 07:57:40.504247 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 5 I0506 07:57:40.505946 11863 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.505063 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.505342 (+ 279us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.505494 (+ 152us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":80} W0506 07:57:40.506310 11855 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:50112: Network error: Client connection negotiation failed: client connection to 127.0.196.124:50112: connect: Connection refused (error 111) I0506 07:57:40.547705 12113 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.124:50112 I0506 07:57:40.548640 12113 webserver.cc:173] Starting webserver on 127.0.196.124:0 I0506 07:57:40.548774 12113 webserver.cc:184] Document root disabled I0506 07:57:40.550967 12113 webserver.cc:311] Webserver started. Bound to: http://127.0.196.124:39269/ I0506 07:57:40.551749 12113 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/master-2/data/info.pb I0506 07:57:40.556484 12241 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0506 07:57:40.562310 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 12113 I0506 07:57:40.569550 12241 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:40.593763 12106 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 5 I0506 07:57:40.604140 12244 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.573168 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56435 0506 07:57:40.576423 (+ 3255us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:40.576446 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:40.576484 (+ 38us) server_negotiation.cc:373] Connection header received 0506 07:57:40.580262 (+ 3778us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:40.580285 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:40.580399 (+ 114us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:40.580530 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:40.582562 (+ 2032us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.583283 (+ 721us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.584509 (+ 1226us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.585377 (+ 868us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.585532 (+ 155us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.588524 (+ 2992us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:40.588540 (+ 16us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:40.588552 (+ 12us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:40.588602 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:40.593503 (+ 4901us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.596353 (+ 2850us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.596377 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.596394 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.597056 (+ 662us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.597744 (+ 688us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.597768 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.597785 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.598225 (+ 440us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:40.598380 (+ 155us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:40.602051 (+ 3671us) server_negotiation.cc:264] Negotiation successful 0506 07:57:40.602461 (+ 410us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3055,"thread_start_us":2797,"threads_started":1} I0506 07:57:40.605959 12243 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.572963 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:40.575539 (+ 2576us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.575642 (+ 103us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:40.579857 (+ 4215us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:40.581138 (+ 1281us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:40.581159 (+ 21us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:40.581919 (+ 760us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:40.582142 (+ 223us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.582380 (+ 238us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.582413 (+ 33us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.583444 (+ 1031us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.583460 (+ 16us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.584327 (+ 867us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.584355 (+ 28us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.585580 (+ 1225us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.585634 (+ 54us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.585951 (+ 317us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.586105 (+ 154us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:40.586118 (+ 13us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:40.588304 (+ 2186us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:40.593738 (+ 5434us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.593769 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.593796 (+ 27us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.596158 (+ 2362us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.597231 (+ 1073us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.597257 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.597277 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.597538 (+ 261us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.598509 (+ 971us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:40.598536 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:40.598964 (+ 428us) client_negotiation.cc:695] Sending connection context 0506 07:57:40.599232 (+ 268us) client_negotiation.cc:225] Negotiation successful 0506 07:57:40.599919 (+ 687us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2250,"thread_start_us":1595,"threads_started":1} I0506 07:57:40.607997 12241 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. I0506 07:57:40.611449 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:40.641455 12112 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.594586 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.595278 (+ 692us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.595311 (+ 33us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:40.595439 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:40.607153 (+ 11714us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:40.607178 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:40.607863 (+ 685us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:40.608016 (+ 153us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.608217 (+ 201us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.608258 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.609773 (+ 1515us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.609795 (+ 22us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.610974 (+ 1179us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.611016 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.613885 (+ 2869us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.613907 (+ 22us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.614239 (+ 332us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.614514 (+ 275us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:40.614530 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:40.617040 (+ 2510us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:40.626855 (+ 9815us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.626880 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.626899 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.633563 (+ 6664us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.635703 (+ 2140us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.635730 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.635748 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.635906 (+ 158us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.637946 (+ 2040us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:40.637972 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:40.638281 (+ 309us) client_negotiation.cc:695] Sending connection context 0506 07:57:40.639112 (+ 831us) client_negotiation.cc:225] Negotiation successful 0506 07:57:40.639664 (+ 552us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":6055} I0506 07:57:40.641953 12246 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.594880 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36839 0506 07:57:40.605907 (+ 11027us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:40.605943 (+ 36us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:40.606002 (+ 59us) server_negotiation.cc:373] Connection header received 0506 07:57:40.606319 (+ 317us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:40.606351 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:40.606917 (+ 566us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:40.607086 (+ 169us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:40.608935 (+ 1849us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.609516 (+ 581us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.611203 (+ 1687us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.613690 (+ 2487us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.614021 (+ 331us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.618788 (+ 4767us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:40.618829 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:40.618861 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:40.618940 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:40.626687 (+ 7747us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.634791 (+ 8104us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.634819 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.634837 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.635511 (+ 674us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.637166 (+ 1655us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.637191 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.637208 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.637772 (+ 564us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:40.637992 (+ 220us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:40.639717 (+ 1725us) server_negotiation.cc:264] Negotiation successful 0506 07:57:40.640179 (+ 462us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":886,"server-negotiator.queue_time_us":10475,"thread_start_us":8888,"threads_started":1} I0506 07:57:40.647480 12243 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.608626 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:40.609073 (+ 447us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.609109 (+ 36us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:40.609290 (+ 181us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:40.610735 (+ 1445us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:40.610759 (+ 24us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:40.611227 (+ 468us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:40.611374 (+ 147us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.611645 (+ 271us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.611686 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.612619 (+ 933us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.612645 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.613975 (+ 1330us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.614015 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.616372 (+ 2357us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.616400 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.617229 (+ 829us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.617375 (+ 146us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:40.617391 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:40.626947 (+ 9556us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:40.635767 (+ 8820us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.635793 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.635813 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.636977 (+ 1164us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.639151 (+ 2174us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.639178 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.639197 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.639350 (+ 153us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.644510 (+ 5160us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:40.644535 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:40.644862 (+ 327us) client_negotiation.cc:695] Sending connection context 0506 07:57:40.645068 (+ 206us) client_negotiation.cc:225] Negotiation successful 0506 07:57:40.645579 (+ 511us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1781} I0506 07:57:40.650745 12247 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.608757 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59130 0506 07:57:40.610327 (+ 1570us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:40.610344 (+ 17us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:40.610368 (+ 24us) server_negotiation.cc:373] Connection header received 0506 07:57:40.610436 (+ 68us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:40.610450 (+ 14us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:40.610526 (+ 76us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:40.610695 (+ 169us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:40.611883 (+ 1188us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.612391 (+ 508us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.614235 (+ 1844us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.615882 (+ 1647us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.616062 (+ 180us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.627230 (+ 11168us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:40.627258 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:40.627280 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:40.627361 (+ 81us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:40.634570 (+ 7209us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.638158 (+ 3588us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.638189 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.638208 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.638963 (+ 755us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.643333 (+ 4370us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.643359 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.643377 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.644316 (+ 939us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:40.648124 (+ 3808us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:40.648331 (+ 207us) server_negotiation.cc:264] Negotiation successful 0506 07:57:40.648831 (+ 500us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":721,"server-negotiator.queue_time_us":1387,"thread_start_us":5457,"threads_started":1} I0506 07:57:40.649292 12106 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: Bootstrap starting. I0506 07:57:40.651999 12241 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:40.654325 12106 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: No blocks or log segments found. Creating new log. I0506 07:57:40.660131 12106 log.cc:520] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: Log is configured to *not* fsync() on all Append() calls May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:40.684118 12243 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.652727 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:40.653284 (+ 557us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:40.653320 (+ 36us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:40.653471 (+ 151us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:40.654527 (+ 1056us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:40.654549 (+ 22us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:40.655205 (+ 656us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:40.655346 (+ 141us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.655546 (+ 200us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.655586 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.656186 (+ 600us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.656212 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.657418 (+ 1206us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:40.657460 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.658649 (+ 1189us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.658676 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:40.659034 (+ 358us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.659127 (+ 93us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:40.659137 (+ 10us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:40.661454 (+ 2317us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:40.673514 (+ 12060us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.673538 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.673556 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.675842 (+ 2286us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.678997 (+ 3155us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:40.679022 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:40.679040 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:40.679260 (+ 220us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:40.681257 (+ 1997us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:40.681278 (+ 21us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:40.681553 (+ 275us) client_negotiation.cc:695] Sending connection context 0506 07:57:40.681777 (+ 224us) client_negotiation.cc:225] Negotiation successful 0506 07:57:40.682274 (+ 497us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":10} I0506 07:57:40.684774 12106 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: No bootstrap required, opened a new log I0506 07:57:40.686712 12246 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:40.653947 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36841 0506 07:57:40.654176 (+ 229us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:40.654190 (+ 14us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:40.654214 (+ 24us) server_negotiation.cc:373] Connection header received 0506 07:57:40.654274 (+ 60us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:40.654288 (+ 14us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:40.654359 (+ 71us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:40.654455 (+ 96us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:40.655763 (+ 1308us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.656054 (+ 291us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.657623 (+ 1569us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:40.658458 (+ 835us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:40.658615 (+ 157us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:40.661717 (+ 3102us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:40.661745 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:40.661768 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:40.661838 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:40.673284 (+ 11446us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.678139 (+ 4855us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.678165 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.678183 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.678834 (+ 651us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:40.680445 (+ 1611us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:40.680472 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:40.680489 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:40.680989 (+ 500us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:40.681150 (+ 161us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:40.684712 (+ 3562us) server_negotiation.cc:264] Negotiation successful 0506 07:57:40.685194 (+ 482us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":66} I0506 07:57:40.694391 12241 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: Bootstrap starting. I0506 07:57:40.697957 12241 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: No blocks or log segments found. Creating new log. I0506 07:57:40.699995 12241 log.cc:520] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: Log is configured to *not* fsync() on all Append() calls I0506 07:57:40.712898 12241 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: No bootstrap required, opened a new log I0506 07:57:40.712661 12106 raft_consensus.cc:303] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:40.713356 12106 raft_consensus.cc:604] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25970fe40aa44c0d8dff72b5a0ace9a7, State: Initialized, Role: FOLLOWER I0506 07:57:40.713820 12106 consensus_queue.cc:227] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:40.724833 12106 sys_catalog.cc:335] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:40.725574 12106 sys_catalog.cc:338] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:40.727017 12106 sys_catalog.cc:422] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:40.728762 12241 raft_consensus.cc:303] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:40.729506 12241 raft_consensus.cc:604] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28567823d1c94558968cb9d87645f6a9, State: Initialized, Role: FOLLOWER I0506 07:57:40.730039 12241 consensus_queue.cc:227] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:40.736661 12259 sys_catalog.cc:335] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:40.749933 12259 sys_catalog.cc:338] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:40.725083 12256 sys_catalog.cc:335] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:40.842995 12256 sys_catalog.cc:338] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:40.847374 12258 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:40.848315 11865 master_main.cc:79] Master server successfully started. I0506 07:57:40.737411 12241 sys_catalog.cc:335] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:40.868804 12241 sys_catalog.cc:338] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:40.870167 12241 sys_catalog.cc:422] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: configured and running, proceeding with master startup. WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:40.888481 12248 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:40.889075 12248 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 I0506 07:57:40.891007 12113 master_main.cc:79] Master server successfully started. W0506 07:57:40.894671 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:40.894924 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:40.893736 12248 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:40.908725 12193 tablet.cc:1678] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: Can't schedule compaction. Clean time has not been advanced past its initial value. W0506 07:57:40.910576 12248 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:40.910809 12248 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:40.911087 12248 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:57:40.911648 12248 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 W0506 07:57:40.925940 12055 tablet.cc:1678] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:40.954372 12248 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:40.969663 12248 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:40.973417 12248 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 225678us May 06 07:57:40 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:40.980036 12248 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:57:40.996742 12248 fs_manager.cc:260] Metadata directory not provided I0506 07:57:40.996968 12248 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:40.997292 12248 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0506 07:57:40.997639 12248 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:41.013381 12248 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "cd4b5cda867b495dbae6ca19be5eee10" format_stamp: "Formatted at 2018-05-06 14:57:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:41.015360 12248 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "cd4b5cda867b495dbae6ca19be5eee10" format_stamp: "Formatted at 2018-05-06 14:57:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:41.043522 12248 fs_manager.cc:495] Time spent creating directory manager: real 0.027s user 0.012s sys 0.008s I0506 07:57:41.045403 12248 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:41.055263 12248 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:41.094877 12248 fs_manager.cc:417] Time spent opening block manager: real 0.015s user 0.001s sys 0.005s I0506 07:57:41.095149 12248 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/wal uuid: "cd4b5cda867b495dbae6ca19be5eee10" format_stamp: "Formatted at 2018-05-06 14:57:41 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:41.095577 12248 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:41.045676 11855 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } attempt: 6 May 06 07:57:41 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:41.192762 12345 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:41.165499 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:41.167138 (+ 1639us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:41.167188 (+ 50us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:41.167456 (+ 268us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:41.168262 (+ 806us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:41.168286 (+ 24us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:41.169542 (+ 1256us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:41.169725 (+ 183us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:41.169929 (+ 204us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:41.169970 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:41.170887 (+ 917us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:41.170916 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:41.172151 (+ 1235us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:41.172192 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:41.174383 (+ 2191us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:41.174409 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:41.174821 (+ 412us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:41.174979 (+ 158us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:41.174997 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:41.179000 (+ 4003us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:41.186002 (+ 7002us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:41.186029 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:41.186049 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:41.187274 (+ 1225us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:41.188332 (+ 1058us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:41.188356 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:41.188374 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:41.188529 (+ 155us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:41.189501 (+ 972us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:41.189524 (+ 23us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:41.189843 (+ 319us) client_negotiation.cc:695] Sending connection context 0506 07:57:41.190050 (+ 207us) client_negotiation.cc:225] Negotiation successful 0506 07:57:41.190563 (+ 513us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":1142,"threads_started":1} I0506 07:57:41.195237 12246 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:41.165704 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36842 0506 07:57:41.165984 (+ 280us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:41.166005 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:41.167280 (+ 1275us) server_negotiation.cc:373] Connection header received 0506 07:57:41.167699 (+ 419us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:41.167724 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:41.167842 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:41.167967 (+ 125us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:41.170152 (+ 2185us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:41.170655 (+ 503us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:41.172390 (+ 1735us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:41.173876 (+ 1486us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:41.174036 (+ 160us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:41.179276 (+ 5240us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:41.179301 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:41.179320 (+ 19us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:41.179386 (+ 66us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:41.185768 (+ 6382us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:41.187469 (+ 1701us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:41.187493 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:41.187510 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:41.188147 (+ 637us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:41.188751 (+ 604us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:41.188774 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:41.188791 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:41.189225 (+ 434us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:41.189384 (+ 159us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:41.193363 (+ 3979us) server_negotiation.cc:264] Negotiation successful 0506 07:57:41.193800 (+ 437us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":82} I0506 07:57:41.202232 11855 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: Bootstrap starting. I0506 07:57:41.205253 11855 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: No blocks or log segments found. Creating new log. I0506 07:57:41.209533 11855 log.cc:520] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: Log is configured to *not* fsync() on all Append() calls I0506 07:57:41.215678 11855 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: No bootstrap required, opened a new log I0506 07:57:41.230584 11855 raft_consensus.cc:303] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:41.231360 11855 raft_consensus.cc:604] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2510444e70934eb4b65ad5cbec6bc5c2, State: Initialized, Role: FOLLOWER I0506 07:57:41.231861 11855 consensus_queue.cc:227] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:41.238107 12366 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:41.238890 11855 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:41.239537 11855 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:41.240850 11855 sys_catalog.cc:422] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:41.242310 12366 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:41.255944 12368 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:41.256939 11750 master_main.cc:79] Master server successfully started. I0506 07:57:41.383245 12248 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:41.394549 12248 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:41.400301 12248 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:41.411573 11770 tablet.cc:1678] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: Can't schedule compaction. Clean time has not been advanced past its initial value. W0506 07:57:41.859894 12258 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:41.860179 12258 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:41.897344 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:41.897575 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:42.255728 12248 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:54327 I0506 07:57:42.261965 12248 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:57:42.262135 12248 webserver.cc:184] Document root disabled I0506 07:57:42.269807 12248 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:35785/ I0506 07:57:42.270815 12248 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-0/data/info.pb W0506 07:57:42.272353 12368 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:42.272634 12368 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:42.273826 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 12248 WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. I0506 07:57:42.326395 12248 tablet_server_main.cc:87] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:57:42.345417 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM May 06 07:57:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:42.436735 12611 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.280336 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:60320 0506 07:57:42.289555 (+ 9219us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:42.289580 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:42.330962 (+ 41382us) server_negotiation.cc:373] Connection header received 0506 07:57:42.366021 (+ 35059us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:42.366052 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:42.366186 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:42.366366 (+ 180us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:42.388180 (+ 21814us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.388759 (+ 579us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.390660 (+ 1901us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.392201 (+ 1541us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.392392 (+ 191us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.403378 (+ 10986us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:42.403457 (+ 79us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:42.403484 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:42.403562 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:42.411918 (+ 8356us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.419046 (+ 7128us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.419074 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.419099 (+ 25us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.419778 (+ 679us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.423065 (+ 3287us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.423090 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.423107 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.423574 (+ 467us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:42.424141 (+ 567us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:42.434775 (+ 10634us) server_negotiation.cc:264] Negotiation successful 0506 07:57:42.435088 (+ 313us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9004,"thread_start_us":9211,"threads_started":1} I0506 07:57:42.440279 12623 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.320734 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:45756 0506 07:57:42.330254 (+ 9520us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:42.330284 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:42.335745 (+ 5461us) server_negotiation.cc:373] Connection header received 0506 07:57:42.375781 (+ 40036us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:42.375809 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:42.375930 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:42.376079 (+ 149us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:42.386688 (+ 10609us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.387214 (+ 526us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.394777 (+ 7563us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.396287 (+ 1510us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.396467 (+ 180us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.404553 (+ 8086us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:42.404650 (+ 97us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:42.404676 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:42.404758 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:42.412568 (+ 7810us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.420566 (+ 7998us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.420620 (+ 54us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.420642 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.423338 (+ 2696us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.435515 (+ 12177us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.435544 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.435564 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.436094 (+ 530us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:42.436484 (+ 390us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:42.438067 (+ 1583us) server_negotiation.cc:264] Negotiation successful 0506 07:57:42.438410 (+ 343us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9250,"thread_start_us":8235,"threads_started":1} I0506 07:57:42.440290 12621 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.319392 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:42.332009 (+ 12617us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:42.332206 (+ 197us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:42.365642 (+ 33436us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:42.376392 (+ 10750us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:42.376426 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:42.377217 (+ 791us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:42.377539 (+ 322us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.386403 (+ 8864us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.386447 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.387880 (+ 1433us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.387908 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.394467 (+ 6559us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.394526 (+ 59us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.397237 (+ 2711us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.397287 (+ 50us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.397763 (+ 476us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.398013 (+ 250us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:42.398033 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:42.403812 (+ 5779us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:42.413387 (+ 9575us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.413423 (+ 36us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.413445 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.415779 (+ 2334us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.425367 (+ 9588us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.425394 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.425415 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.434374 (+ 8959us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.436920 (+ 2546us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:42.436948 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:42.437292 (+ 344us) client_negotiation.cc:695] Sending connection context 0506 07:57:42.437748 (+ 456us) client_negotiation.cc:225] Negotiation successful 0506 07:57:42.438070 (+ 322us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":12292,"mutex_wait_us":14253,"thread_start_us":11508,"threads_started":1} I0506 07:57:42.441390 12614 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.293212 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:42.330674 (+ 37462us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:42.330791 (+ 117us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:42.365641 (+ 34850us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:42.379350 (+ 13709us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:42.381308 (+ 1958us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:42.386289 (+ 4981us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:42.386484 (+ 195us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.387921 (+ 1437us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.387962 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.388971 (+ 1009us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.389000 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.390387 (+ 1387us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.390440 (+ 53us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.392497 (+ 2057us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.392529 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.394979 (+ 2450us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.395285 (+ 306us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:42.395303 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:42.402629 (+ 7326us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:42.412469 (+ 9840us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.412502 (+ 33us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.412527 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.414293 (+ 1766us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.420313 (+ 6020us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.420341 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.420363 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.420617 (+ 254us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.424312 (+ 3695us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:42.424348 (+ 36us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:42.432739 (+ 8391us) client_negotiation.cc:695] Sending connection context 0506 07:57:42.434462 (+ 1723us) client_negotiation.cc:225] Negotiation successful 0506 07:57:42.434982 (+ 520us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":36078,"mutex_wait_us":5311,"spinlock_wait_cycles":26368,"thread_start_us":8586,"threads_started":1} I0506 07:57:42.445101 12609 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51652 I0506 07:57:42.445531 12609 heartbeater.cc:426] Registering TS with master... I0506 07:57:42.446295 12609 heartbeater.cc:475] Master 127.0.196.126:51652 requested a full tablet report, sending... I0506 07:57:42.449339 11781 ts_manager.cc:81] Registered new tserver with Master: cd4b5cda867b495dbae6ca19be5eee10 (127.0.196.65:54327) I0506 07:57:42.458840 12622 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.315882 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:42.330786 (+ 14904us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:42.332382 (+ 1596us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:42.365790 (+ 33408us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:42.368275 (+ 2485us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:42.368310 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:42.369299 (+ 989us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:42.369661 (+ 362us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.370038 (+ 377us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.370087 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.373800 (+ 3713us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.373828 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.375368 (+ 1540us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:42.375414 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.419937 (+ 44523us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.419972 (+ 35us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:42.421448 (+ 1476us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.421720 (+ 272us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:42.421740 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:42.424576 (+ 2836us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:42.432478 (+ 7902us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.432515 (+ 37us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.432543 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.433827 (+ 1284us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.451770 (+ 17943us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:42.451798 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:42.451821 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:42.451993 (+ 172us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:42.455799 (+ 3806us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:42.455824 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:42.456117 (+ 293us) client_negotiation.cc:695] Sending connection context 0506 07:57:42.456502 (+ 385us) client_negotiation.cc:225] Negotiation successful 0506 07:57:42.456835 (+ 333us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":14577,"mutex_wait_us":1005,"thread_start_us":16458,"threads_started":1} I0506 07:57:42.465277 12619 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:42.315204 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:35781 0506 07:57:42.316609 (+ 1405us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:42.316634 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:42.332474 (+ 15840us) server_negotiation.cc:373] Connection header received 0506 07:57:42.366765 (+ 34291us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:42.366791 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:42.366902 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:42.367039 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:42.370281 (+ 3242us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.370850 (+ 569us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.375930 (+ 5080us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:42.377460 (+ 1530us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:42.377644 (+ 184us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:42.425251 (+ 47607us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:42.425333 (+ 82us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:42.425359 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:42.425443 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:42.431912 (+ 6469us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.442392 (+ 10480us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.442420 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.442440 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.451080 (+ 8640us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:42.453928 (+ 2848us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:42.453952 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:42.453971 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:42.455143 (+ 1172us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:42.462715 (+ 7572us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:42.463167 (+ 452us) server_negotiation.cc:264] Negotiation successful 0506 07:57:42.463493 (+ 326us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":969,"threads_started":1} I0506 07:57:42.445693 12616 heartbeater.cc:346] Connected to a master server at 127.0.196.124:50112 I0506 07:57:42.465788 12616 heartbeater.cc:426] Registering TS with master... I0506 07:57:42.466301 12616 heartbeater.cc:475] Master 127.0.196.124:50112 requested a full tablet report, sending... I0506 07:57:42.468478 12613 heartbeater.cc:346] Connected to a master server at 127.0.196.125:38563 I0506 07:57:42.468741 12613 heartbeater.cc:426] Registering TS with master... I0506 07:57:42.469198 12613 heartbeater.cc:475] Master 127.0.196.125:38563 requested a full tablet report, sending... I0506 07:57:42.469328 12204 ts_manager.cc:81] Registered new tserver with Master: cd4b5cda867b495dbae6ca19be5eee10 (127.0.196.65:54327) I0506 07:57:42.471930 12070 ts_manager.cc:81] Registered new tserver with Master: cd4b5cda867b495dbae6ca19be5eee10 (127.0.196.65:54327) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:42.746917 12627 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:42.752323 12627 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:57:42.757035 12627 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:42.785344 12627 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:42.793736 12627 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:42.794178 12627 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:57:42.794802 12627 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 W0506 07:57:42.862629 12258 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:42.862862 12258 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:42.863620 12627 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:42.887847 12627 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:42.892191 12627 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 226678us May 06 07:57:42 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:42.897846 12627 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) W0506 07:57:42.899976 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:42.900254 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:42.927090 12627 fs_manager.cc:260] Metadata directory not provided I0506 07:57:42.927320 12627 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:42.927695 12627 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:57:42.927963 12627 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:42.960098 12627 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "85ae14bafd2c430cb1c14e0af50d2aa8" format_stamp: "Formatted at 2018-05-06 14:57:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:42.961279 12627 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "85ae14bafd2c430cb1c14e0af50d2aa8" format_stamp: "Formatted at 2018-05-06 14:57:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:43.010699 12627 fs_manager.cc:495] Time spent creating directory manager: real 0.041s user 0.014s sys 0.004s I0506 07:57:43.013751 12627 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:43.014433 12627 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:43.037863 12627 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s I0506 07:57:43.038429 12627 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/wal uuid: "85ae14bafd2c430cb1c14e0af50d2aa8" format_stamp: "Formatted at 2018-05-06 14:57:42 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:43.039222 12627 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0506 07:57:43.275095 12368 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:43.275306 12368 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:43.622213 12627 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:43.628464 12627 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:43.637552 12627 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:43.865205 12258 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:43.865404 12258 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:43.902669 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:43.902873 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:44.277688 12368 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:44.277900 12368 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:44.509613 12627 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:38605 I0506 07:57:44.511420 12627 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:57:44.511616 12627 webserver.cc:184] Document root disabled I0506 07:57:44.522943 12627 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:60766/ I0506 07:57:44.524044 12627 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:57:44.530274 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 12627 I0506 07:57:44.577366 12627 tablet_server_main.cc:87] Tablet server successfully started. WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. W0506 07:57:44.635901 785 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.196.67: real 0.105s user 0.001s sys 0.010s I0506 07:57:44.637547 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:44 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:44 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:44 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:44.689847 12922 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.536084 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:54781 0506 07:57:44.552138 (+ 16054us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:44.552164 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:44.552204 (+ 40us) server_negotiation.cc:373] Connection header received 0506 07:57:44.627075 (+ 74871us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:44.627103 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:44.627233 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:44.627381 (+ 148us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:44.632309 (+ 4928us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.632880 (+ 571us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.636912 (+ 4032us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.638385 (+ 1473us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.638560 (+ 175us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.652963 (+ 14403us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:44.653041 (+ 78us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:44.653069 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:44.653158 (+ 89us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:44.661542 (+ 8384us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.673194 (+ 11652us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.673221 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.673242 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.673953 (+ 711us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.676731 (+ 2778us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.676758 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.676778 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.677255 (+ 477us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:44.677615 (+ 360us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:44.683223 (+ 5608us) server_negotiation.cc:264] Negotiation successful 0506 07:57:44.683559 (+ 336us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15841,"thread_start_us":17438,"threads_started":1} I0506 07:57:44.690284 12927 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.549567 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:44.553320 (+ 3753us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:44.553394 (+ 74us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:44.626385 (+ 72991us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:44.627442 (+ 1057us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:44.627477 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:44.636883 (+ 9406us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:44.637177 (+ 294us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.637761 (+ 584us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.637805 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.639048 (+ 1243us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.639076 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.640660 (+ 1584us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.640727 (+ 67us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.645040 (+ 4313us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.645067 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.645625 (+ 558us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.645906 (+ 281us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:44.645923 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:44.654761 (+ 8838us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:44.671076 (+ 16315us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.671103 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.671125 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.672434 (+ 1309us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.679142 (+ 6708us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.679169 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.679192 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.679355 (+ 163us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.685733 (+ 6378us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:44.685760 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:44.686050 (+ 290us) client_negotiation.cc:695] Sending connection context 0506 07:57:44.686459 (+ 409us) client_negotiation.cc:225] Negotiation successful 0506 07:57:44.686870 (+ 411us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3405,"mutex_wait_us":5552,"thread_start_us":2985,"threads_started":1} I0506 07:57:44.694764 12924 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.536399 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:44.544332 (+ 7933us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:44.544442 (+ 110us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:44.626385 (+ 81943us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:44.627744 (+ 1359us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:44.627777 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:44.628680 (+ 903us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:44.630603 (+ 1923us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.631204 (+ 601us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.631251 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.633298 (+ 2047us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.633327 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.635794 (+ 2467us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.635843 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.639358 (+ 3515us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.639390 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.640217 (+ 827us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.641829 (+ 1612us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:44.641852 (+ 23us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:44.649350 (+ 7498us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:44.663049 (+ 13699us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.663080 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.663107 (+ 27us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.669939 (+ 6832us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.675130 (+ 5191us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.675159 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.675180 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.675447 (+ 267us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.678595 (+ 3148us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:44.678626 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:44.681313 (+ 2687us) client_negotiation.cc:695] Sending connection context 0506 07:57:44.691182 (+ 9869us) client_negotiation.cc:225] Negotiation successful 0506 07:57:44.691698 (+ 516us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7260,"mutex_wait_us":886,"thread_start_us":3500,"threads_started":1} I0506 07:57:44.699510 12920 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51652 I0506 07:57:44.699854 12920 heartbeater.cc:426] Registering TS with master... I0506 07:57:44.700462 12920 heartbeater.cc:475] Master 127.0.196.126:51652 requested a full tablet report, sending... I0506 07:57:44.703018 12930 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.560823 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:44.574718 (+ 13895us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:44.574803 (+ 85us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:44.627891 (+ 53088us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:44.628881 (+ 990us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:44.628919 (+ 38us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:44.637128 (+ 8209us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:44.637381 (+ 253us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.646141 (+ 8760us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.646182 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.647219 (+ 1037us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.647248 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.649105 (+ 1857us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:44.649152 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.651123 (+ 1971us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.651152 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:44.651591 (+ 439us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.651917 (+ 326us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:44.651938 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:44.657866 (+ 5928us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:44.679341 (+ 21475us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.679368 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.679388 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.681317 (+ 1929us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.683768 (+ 2451us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:44.683796 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:44.683817 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:44.683979 (+ 162us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:44.690806 (+ 6827us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:44.690835 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:44.691170 (+ 335us) client_negotiation.cc:695] Sending connection context 0506 07:57:44.691687 (+ 517us) client_negotiation.cc:225] Negotiation successful 0506 07:57:44.692000 (+ 313us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13552,"mutex_wait_us":4655,"thread_start_us":12191,"threads_started":1} I0506 07:57:44.703284 11781 ts_manager.cc:81] Registered new tserver with Master: 85ae14bafd2c430cb1c14e0af50d2aa8 (127.0.196.66:38605) I0506 07:57:44.703614 12926 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.550059 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:49224 0506 07:57:44.562553 (+ 12494us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:44.562582 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:44.562620 (+ 38us) server_negotiation.cc:373] Connection header received 0506 07:57:44.626846 (+ 64226us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:44.626875 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:44.627005 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:44.627185 (+ 180us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:44.638017 (+ 10832us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.638561 (+ 544us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.641887 (+ 3326us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.643355 (+ 1468us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.643517 (+ 162us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.663055 (+ 19538us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:44.663129 (+ 74us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:44.663159 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:44.663244 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:44.670462 (+ 7218us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.677950 (+ 7488us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.677978 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.677998 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.678650 (+ 652us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.684676 (+ 6026us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.684705 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.684725 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.685196 (+ 471us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:44.685585 (+ 389us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:44.689510 (+ 3925us) server_negotiation.cc:264] Negotiation successful 0506 07:57:44.689855 (+ 345us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12286,"thread_start_us":13637,"threads_started":1} I0506 07:57:44.704883 12929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:44.560726 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:50795 0506 07:57:44.572130 (+ 11404us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:44.572155 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:44.574894 (+ 2739us) server_negotiation.cc:373] Connection header received 0506 07:57:44.628181 (+ 53287us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:44.628212 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:44.628340 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:44.628486 (+ 146us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:44.646399 (+ 17913us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.646925 (+ 526us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.649403 (+ 2478us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:44.650892 (+ 1489us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:44.651071 (+ 179us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:44.658624 (+ 7553us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:44.658700 (+ 76us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:44.658726 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:44.658806 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:44.673654 (+ 14848us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.682375 (+ 8721us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.682401 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.682420 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.683086 (+ 666us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:44.688160 (+ 5074us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:44.688187 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:44.688205 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:44.688736 (+ 531us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:44.693316 (+ 4580us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:44.693969 (+ 653us) server_negotiation.cc:264] Negotiation successful 0506 07:57:44.694304 (+ 335us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11205,"thread_start_us":11387,"threads_started":1} I0506 07:57:44.706171 12923 heartbeater.cc:346] Connected to a master server at 127.0.196.125:38563 I0506 07:57:44.706413 12923 heartbeater.cc:426] Registering TS with master... I0506 07:57:44.706895 12923 heartbeater.cc:475] Master 127.0.196.125:38563 requested a full tablet report, sending... I0506 07:57:44.709359 12070 ts_manager.cc:81] Registered new tserver with Master: 85ae14bafd2c430cb1c14e0af50d2aa8 (127.0.196.66:38605) I0506 07:57:44.707818 12925 heartbeater.cc:346] Connected to a master server at 127.0.196.124:50112 I0506 07:57:44.718582 12925 heartbeater.cc:426] Registering TS with master... I0506 07:57:44.719069 12925 heartbeater.cc:475] Master 127.0.196.124:50112 requested a full tablet report, sending... I0506 07:57:44.721470 12204 ts_manager.cc:81] Registered new tserver with Master: 85ae14bafd2c430cb1c14e0af50d2aa8 (127.0.196.66:38605) W0506 07:57:44.867740 12258 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:44.867954 12258 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:44.904446 12932 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:44.904908 12932 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:57:44.905187 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:44.905388 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:44.905408 12932 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:44.921456 12932 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:44.922165 12932 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:44.922461 12932 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:57:44.923025 12932 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:57:44.957031 12932 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:51652,127.0.196.125:38563,127.0.196.124:50112 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:44.972851 12932 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:44.975239 12932 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 227678us May 06 07:57:44 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:44.978471 12932 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:57:44.991118 12932 fs_manager.cc:260] Metadata directory not provided I0506 07:57:44.991320 12932 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:44.991703 12932 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:57:44.991986 12932 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:45.008900 12932 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "e6157d4018a54771a1aa77ed085b18dd" format_stamp: "Formatted at 2018-05-06 14:57:45 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:45.010138 12932 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "e6157d4018a54771a1aa77ed085b18dd" format_stamp: "Formatted at 2018-05-06 14:57:45 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:45.029547 12932 fs_manager.cc:495] Time spent creating directory manager: real 0.019s user 0.012s sys 0.003s I0506 07:57:45.031288 12932 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:45.031903 12932 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:45.048768 12932 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s I0506 07:57:45.048950 12932 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/wal uuid: "e6157d4018a54771a1aa77ed085b18dd" format_stamp: "Formatted at 2018-05-06 14:57:45 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:45.049271 12932 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0506 07:57:45.280239 12368 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:45.280445 12368 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:45.579972 12932 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:45.589555 12932 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:45.611500 12932 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:45.870359 12258 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:45.870569 12258 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:45.907666 12275 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:45.907871 12275 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:46.246330 12932 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:54374 I0506 07:57:46.247985 12932 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:57:46.248167 12932 webserver.cc:184] Document root disabled I0506 07:57:46.259667 12932 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:44789/ I0506 07:57:46.260820 12932 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:57:46.269551 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 12932 W0506 07:57:46.282796 12368 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:46.283043 12368 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:46.313999 12932 tablet_server_main.cc:87] Tablet server successfully started. I0506 07:57:46.327889 13314 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.270911 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36856 0506 07:57:46.276765 (+ 5854us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.276789 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.281220 (+ 4431us) server_negotiation.cc:373] Connection header received 0506 07:57:46.281341 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.281365 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.281484 (+ 119us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.281650 (+ 166us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.290805 (+ 9155us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.291356 (+ 551us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.293353 (+ 1997us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.294907 (+ 1554us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.295084 (+ 177us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.300434 (+ 5350us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.300463 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.300487 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.300560 (+ 73us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.311654 (+ 11094us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.320300 (+ 8646us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.320325 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.320341 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.320993 (+ 652us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.321764 (+ 771us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.321790 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.321809 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.322282 (+ 473us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.322626 (+ 344us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.325744 (+ 3118us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.326174 (+ 430us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5628,"thread_start_us":8636,"threads_started":1} May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:46.412827 13313 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.265566 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:36106 0506 07:57:46.275583 (+ 10017us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.275633 (+ 50us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.285513 (+ 9880us) server_negotiation.cc:373] Connection header received 0506 07:57:46.330141 (+ 44628us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.330168 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.330286 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.330422 (+ 136us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.357678 (+ 27256us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.358241 (+ 563us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.360683 (+ 2442us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.362205 (+ 1522us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.362373 (+ 168us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.384946 (+ 22573us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.385021 (+ 75us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.385049 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.385137 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.392503 (+ 7366us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.399469 (+ 6966us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.399496 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.399515 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.400574 (+ 1059us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.403330 (+ 2756us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.403356 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.403374 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.404244 (+ 870us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.404588 (+ 344us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.408690 (+ 4102us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.409026 (+ 336us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9812,"thread_start_us":4654,"threads_started":1} I0506 07:57:46.414706 13322 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.287395 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:50112 0506 07:57:46.298568 (+ 11173us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:46.298614 (+ 46us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:46.329358 (+ 30744us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:46.330329 (+ 971us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:46.330364 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:46.331845 (+ 1481us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:46.346500 (+ 14655us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.348447 (+ 1947us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.348488 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.349776 (+ 1288us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.349801 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.356506 (+ 6705us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.357361 (+ 855us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.370101 (+ 12740us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.370135 (+ 34us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.370553 (+ 418us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.371989 (+ 1436us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:46.372008 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:46.383214 (+ 11206us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:46.392708 (+ 9494us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.392736 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.392767 (+ 31us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.396389 (+ 3622us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.400425 (+ 4036us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.400456 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.400478 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.400797 (+ 319us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.410897 (+ 10100us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:46.410929 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:46.411225 (+ 296us) client_negotiation.cc:695] Sending connection context 0506 07:57:46.411651 (+ 426us) client_negotiation.cc:225] Negotiation successful 0506 07:57:46.412104 (+ 453us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1435,"mutex_wait_us":11674,"thread_start_us":1558,"threads_started":1} I0506 07:57:46.419117 13323 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.288373 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:57952 0506 07:57:46.289851 (+ 1478us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.289873 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.298893 (+ 9020us) server_negotiation.cc:373] Connection header received 0506 07:57:46.329558 (+ 30665us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.329584 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.329737 (+ 153us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.329871 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.348679 (+ 18808us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.349206 (+ 527us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.357559 (+ 8353us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.359132 (+ 1573us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.359304 (+ 172us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.384174 (+ 24870us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.384249 (+ 75us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.384274 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.384353 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.391363 (+ 7010us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.398944 (+ 7581us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.399055 (+ 111us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.399078 (+ 23us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.399765 (+ 687us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.409526 (+ 9761us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.409556 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.409575 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.410093 (+ 518us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.410475 (+ 382us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.416878 (+ 6403us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.417213 (+ 335us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":1077,"threads_started":1} I0506 07:57:46.421533 13316 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.271326 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:51652 0506 07:57:46.285318 (+ 13992us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:46.285400 (+ 82us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:46.328976 (+ 43576us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:46.330744 (+ 1768us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:46.331149 (+ 405us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:46.349123 (+ 17974us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:46.349304 (+ 181us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.350135 (+ 831us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.350175 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.358854 (+ 8679us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.358890 (+ 36us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.360424 (+ 1534us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.360467 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.363092 (+ 2625us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.363120 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.363537 (+ 417us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.371824 (+ 8287us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:46.371846 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:46.384137 (+ 12291us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:46.393716 (+ 9579us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.393745 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.393766 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.398737 (+ 4971us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.401998 (+ 3261us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.402023 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.402042 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.402189 (+ 147us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.405762 (+ 3573us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:46.405791 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:46.407257 (+ 1466us) client_negotiation.cc:695] Sending connection context 0506 07:57:46.407895 (+ 638us) client_negotiation.cc:225] Negotiation successful 0506 07:57:46.415547 (+ 7652us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13516,"mutex_wait_us":7501,"thread_start_us":12975,"threads_started":1} I0506 07:57:46.425442 13335 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.336550 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59148 0506 07:57:46.351889 (+ 15339us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.351916 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.351953 (+ 37us) server_negotiation.cc:373] Connection header received 0506 07:57:46.352048 (+ 95us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.352073 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.352185 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.352368 (+ 183us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.355311 (+ 2943us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.359465 (+ 4154us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.361321 (+ 1856us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.363216 (+ 1895us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.363402 (+ 186us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.367733 (+ 4331us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.367765 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.367790 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.367863 (+ 73us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.408761 (+ 40898us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.413209 (+ 4448us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.413236 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.413255 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.413935 (+ 680us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.414563 (+ 628us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.414588 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.414609 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.420552 (+ 5943us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.423014 (+ 2462us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.423216 (+ 202us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.423675 (+ 459us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15068,"thread_start_us":6061,"threads_started":1} I0506 07:57:46.430621 13321 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.286566 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:58626 0506 07:57:46.299190 (+ 12624us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.299238 (+ 48us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.299278 (+ 40us) server_negotiation.cc:373] Connection header received 0506 07:57:46.329361 (+ 30083us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.329390 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.329511 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.329676 (+ 165us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.351461 (+ 21785us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.352109 (+ 648us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.354393 (+ 2284us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.356009 (+ 1616us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.356184 (+ 175us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.375489 (+ 19305us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.375608 (+ 119us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.375637 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.375717 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.391304 (+ 15587us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.397108 (+ 5804us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.397136 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.397155 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.397842 (+ 687us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.415811 (+ 17969us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.415841 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.415861 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.422751 (+ 6890us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.423112 (+ 361us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.428320 (+ 5208us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.428698 (+ 378us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":14995,"server-negotiator.queue_time_us":12410,"thread_start_us":12163,"threads_started":1} I0506 07:57:46.430544 13324 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.289323 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:38563 0506 07:57:46.298004 (+ 8681us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:46.298098 (+ 94us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:46.328981 (+ 30883us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:46.332602 (+ 3621us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:46.340607 (+ 8005us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:46.347051 (+ 6444us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:46.347356 (+ 305us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.347765 (+ 409us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.347808 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.352644 (+ 4836us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.352697 (+ 53us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.354109 (+ 1412us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:46.354154 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.356647 (+ 2493us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.356678 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:46.357109 (+ 431us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.357652 (+ 543us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:46.357671 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:46.374858 (+ 17187us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:46.392635 (+ 17777us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.392673 (+ 38us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.392701 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.394747 (+ 2046us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.398501 (+ 3754us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:46.398528 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:46.398548 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:46.401402 (+ 2854us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:46.427242 (+ 25840us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:46.427268 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:46.427553 (+ 285us) client_negotiation.cc:695] Sending connection context 0506 07:57:46.427981 (+ 428us) client_negotiation.cc:225] Negotiation successful 0506 07:57:46.428330 (+ 349us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":8342,"mutex_wait_us":2599,"thread_start_us":7856,"threads_started":1} I0506 07:57:46.439821 13317 heartbeater.cc:346] Connected to a master server at 127.0.196.125:38563 I0506 07:57:46.440194 13317 heartbeater.cc:426] Registering TS with master... I0506 07:57:46.440912 13317 heartbeater.cc:475] Master 127.0.196.125:38563 requested a full tablet report, sending... I0506 07:57:46.442908 13311 heartbeater.cc:346] Connected to a master server at 127.0.196.126:51652 I0506 07:57:46.443158 13311 heartbeater.cc:426] Registering TS with master... I0506 07:57:46.443601 13311 heartbeater.cc:475] Master 127.0.196.126:51652 requested a full tablet report, sending... I0506 07:57:46.444054 12070 ts_manager.cc:81] Registered new tserver with Master: e6157d4018a54771a1aa77ed085b18dd (127.0.196.67:54374) I0506 07:57:46.447432 11781 ts_manager.cc:81] Registered new tserver with Master: e6157d4018a54771a1aa77ed085b18dd (127.0.196.67:54374) May 06 07:57:46 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[11720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:46.473860 13313 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.431274 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56456 0506 07:57:46.438840 (+ 7566us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.438865 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.438904 (+ 39us) server_negotiation.cc:373] Connection header received 0506 07:57:46.439013 (+ 109us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.439037 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.439163 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.439294 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.445336 (+ 6042us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.446208 (+ 872us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.447960 (+ 1752us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.450538 (+ 2578us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.450749 (+ 211us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.455702 (+ 4953us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.455728 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.455793 (+ 65us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.455873 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.465698 (+ 9825us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.467492 (+ 1794us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.467519 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.467538 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.468468 (+ 930us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.469074 (+ 606us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.469108 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.469126 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.469596 (+ 470us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.471620 (+ 2024us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.471844 (+ 224us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.472257 (+ 413us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7383} I0506 07:57:46.422277 13319 heartbeater.cc:346] Connected to a master server at 127.0.196.124:50112 I0506 07:57:46.479485 13319 heartbeater.cc:426] Registering TS with master... I0506 07:57:46.480005 13319 heartbeater.cc:475] Master 127.0.196.124:50112 requested a full tablet report, sending... I0506 07:57:46.482950 12204 ts_manager.cc:81] Registered new tserver with Master: e6157d4018a54771a1aa77ed085b18dd (127.0.196.67:54374) I0506 07:57:46.485131 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0506 07:57:46.486934 11814 tablet_service.cc:1113] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000" dest_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.196.126:56456 I0506 07:57:46.487396 11814 raft_consensus.cc:435] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0506 07:57:46.487650 11814 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:46.492063 11814 raft_consensus.cc:457] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:46.493634 11814 leader_election.cc:230] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [CANDIDATE]: Term 1 election: Requesting vote from peer 25970fe40aa44c0d8dff72b5a0ace9a7 I0506 07:57:46.495622 12080 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" I0506 07:57:46.496204 12080 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:46.501201 12080 raft_consensus.cc:2219] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2510444e70934eb4b65ad5cbec6bc5c2 in term 1. I0506 07:57:46.502141 11762 leader_election.cc:362] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [CANDIDATE]: Term 1 election: Vote granted by peer 25970fe40aa44c0d8dff72b5a0ace9a7 I0506 07:57:46.502378 11762 leader_election.cc:257] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:46.502795 11814 leader_election.cc:230] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [CANDIDATE]: Term 1 election: Requesting vote from peer 28567823d1c94558968cb9d87645f6a9 I0506 07:57:46.504181 12216 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28567823d1c94558968cb9d87645f6a9" I0506 07:57:46.504753 12216 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:46.509385 12216 raft_consensus.cc:2219] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2510444e70934eb4b65ad5cbec6bc5c2 in term 1. I0506 07:57:46.510349 13352 raft_consensus.cc:2511] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:46.510493 11761 leader_election.cc:362] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [CANDIDATE]: Term 1 election: Vote granted by peer 28567823d1c94558968cb9d87645f6a9 I0506 07:57:46.512792 13352 raft_consensus.cc:571] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [term 1 LEADER]: Becoming Leader. State: Replica: 2510444e70934eb4b65ad5cbec6bc5c2, State: Running, Role: LEADER I0506 07:57:46.513254 13352 consensus_queue.cc:204] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } I0506 07:57:46.520519 13354 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2510444e70934eb4b65ad5cbec6bc5c2. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.521159 13354 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: This master's current role is: LEADER I0506 07:57:46.523931 13362 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:46.534814 13362 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:46.604707 12216 raft_consensus.cc:1057] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [term 1 FOLLOWER]: Refusing update from remote peer 2510444e70934eb4b65ad5cbec6bc5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:46.606827 12080 raft_consensus.cc:1057] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [term 1 FOLLOWER]: Refusing update from remote peer 2510444e70934eb4b65ad5cbec6bc5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:46.606775 13354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:46.608424 13354 consensus_queue.cc:963] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:46.636261 13323 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.525971 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36864 0506 07:57:46.526300 (+ 329us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.526323 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.540636 (+ 14313us) server_negotiation.cc:373] Connection header received 0506 07:57:46.541839 (+ 1203us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.541864 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.541986 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.542112 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.547854 (+ 5742us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.548379 (+ 525us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.558341 (+ 9962us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.559896 (+ 1555us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.560073 (+ 177us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.585789 (+ 25716us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.585816 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.585838 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.585913 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.595401 (+ 9488us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.598813 (+ 3412us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.598840 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.598858 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.599518 (+ 660us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.629778 (+ 30260us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.629812 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.629831 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.630431 (+ 600us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.630788 (+ 357us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.634071 (+ 3283us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.634571 (+ 500us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":121} I0506 07:57:46.637946 13313 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.524876 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56460 0506 07:57:46.525154 (+ 278us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.525176 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.556915 (+ 31739us) server_negotiation.cc:373] Connection header received 0506 07:57:46.557040 (+ 125us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.557066 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.557192 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.557318 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.559891 (+ 2573us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.560416 (+ 525us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.564383 (+ 3967us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.565890 (+ 1507us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.566076 (+ 186us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.595122 (+ 29046us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.595151 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.595174 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.595260 (+ 86us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.607810 (+ 12550us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.617784 (+ 9974us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.617815 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.617832 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.618813 (+ 981us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.630918 (+ 12105us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.630945 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.630965 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.631500 (+ 535us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.631816 (+ 316us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.632771 (+ 955us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.633416 (+ 645us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":89} I0506 07:57:46.643745 13382 sys_catalog.cc:335] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2510444e70934eb4b65ad5cbec6bc5c2. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.644383 13382 sys_catalog.cc:338] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:46.656752 13382 sys_catalog.cc:335] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.657414 13382 sys_catalog.cc:338] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:46.658605 13352 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.659335 13352 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: This master's current role is: LEADER I0506 07:57:46.661223 13362 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:46.662994 13362 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:46.676614 13383 sys_catalog.cc:335] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2510444e70934eb4b65ad5cbec6bc5c2. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.676759 13352 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.677335 13383 sys_catalog.cc:338] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:46.677378 13352 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2 [sys.catalog]: This master's current role is: LEADER I0506 07:57:46.651571 13321 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.539383 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59155 0506 07:57:46.539720 (+ 337us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.539743 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.557778 (+ 18035us) server_negotiation.cc:373] Connection header received 0506 07:57:46.562272 (+ 4494us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.562299 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.562411 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.562538 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.583711 (+ 21173us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.584211 (+ 500us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.601577 (+ 17366us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.603146 (+ 1569us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.603324 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.622888 (+ 19564us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:46.622914 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:46.622935 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:46.623005 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:46.637268 (+ 14263us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.643191 (+ 5923us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.643222 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.643243 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.644030 (+ 787us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:46.644838 (+ 808us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:46.644866 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:46.644885 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:46.645384 (+ 499us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:46.645680 (+ 296us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.648477 (+ 2797us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.648995 (+ 518us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":120} I0506 07:57:46.690048 13362 catalog_manager.cc:3992] T 00000000000000000000000000000000 P 2510444e70934eb4b65ad5cbec6bc5c2: Generated new TSK 0 I0506 07:57:46.701177 13383 sys_catalog.cc:335] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2510444e70934eb4b65ad5cbec6bc5c2" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 51652 } } peers { permanent_uuid: "25970fe40aa44c0d8dff72b5a0ace9a7" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 38563 } } peers { permanent_uuid: "28567823d1c94558968cb9d87645f6a9" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 50112 } } } I0506 07:57:46.701833 13383 sys_catalog.cc:338] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:46.715209 11781 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:54781 W0506 07:57:46.838551 13323 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:46.819490 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36867 0506 07:57:46.819805 (+ 315us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.819829 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.829647 (+ 9818us) server_negotiation.cc:373] Connection header received 0506 07:57:46.829947 (+ 300us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.829973 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.830095 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.830214 (+ 119us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.837782 (+ 7568us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:36867: BlockingRecv error: recv got EOF from 127.0.196.124:36867 (error 108) Metrics: {"server-negotiator.queue_time_us":84} W0506 07:57:46.844440 13313 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:46.826299 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56465 0506 07:57:46.826683 (+ 384us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.826705 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.841168 (+ 14463us) server_negotiation.cc:373] Connection header received 0506 07:57:46.841468 (+ 300us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.841494 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.841639 (+ 145us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.841769 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.843677 (+ 1908us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.126:56465: BlockingRecv error: recv got EOF from 127.0.196.126:56465 (error 108) Metrics: {"server-negotiator.queue_time_us":164} W0506 07:57:46.850289 13321 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:46.830503 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59159 0506 07:57:46.830815 (+ 312us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.830838 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.846994 (+ 16156us) server_negotiation.cc:373] Connection header received 0506 07:57:46.847115 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.847142 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.847267 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.847408 (+ 141us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.849528 (+ 2120us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.125:59159: BlockingRecv error: recv got EOF from 127.0.196.125:59159 (error 108) Metrics: {"server-negotiator.queue_time_us":78} I0506 07:57:46.880137 12258 catalog_manager.cc:1034] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: acquiring CA information for follower catalog manager: success W0506 07:57:46.884227 13321 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:46.870874 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59161 0506 07:57:46.871156 (+ 282us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.871180 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.881080 (+ 9900us) server_negotiation.cc:373] Connection header received 0506 07:57:46.881359 (+ 279us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.881376 (+ 17us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.881488 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.881600 (+ 112us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:46.883732 (+ 2132us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.125:59161: BlockingRecv error: recv got EOF from 127.0.196.125:59161 (error 108) Metrics: {"server-negotiator.queue_time_us":86} I0506 07:57:46.886770 12258 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 25970fe40aa44c0d8dff72b5a0ace9a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0506 07:57:46.914882 12275 catalog_manager.cc:1034] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: acquiring CA information for follower catalog manager: success I0506 07:57:46.935667 12275 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 28567823d1c94558968cb9d87645f6a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0506 07:57:46.940583 13321 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.914896 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59162 0506 07:57:46.915245 (+ 349us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.915267 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:46.924975 (+ 9708us) server_negotiation.cc:373] Connection header received 0506 07:57:46.925559 (+ 584us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:46.925585 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:46.925782 (+ 197us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:46.925935 (+ 153us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:46.931658 (+ 5723us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.932234 (+ 576us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.934793 (+ 2559us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:46.936227 (+ 1434us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:46.936395 (+ 168us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:46.937747 (+ 1352us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:46.938204 (+ 457us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:46.938332 (+ 128us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:46.939174 (+ 842us) server_negotiation.cc:264] Negotiation successful 0506 07:57:46.939362 (+ 188us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":132} I0506 07:57:47.027148 13323 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:46.994988 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36873 0506 07:57:46.995335 (+ 347us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:46.995360 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:47.012198 (+ 16838us) server_negotiation.cc:373] Connection header received 0506 07:57:47.012532 (+ 334us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:47.012560 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:47.012766 (+ 206us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:47.012915 (+ 149us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:47.017857 (+ 4942us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.018457 (+ 600us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.021289 (+ 2832us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.022790 (+ 1501us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.022971 (+ 181us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:47.024269 (+ 1298us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:47.024702 (+ 433us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:47.024818 (+ 116us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:47.025693 (+ 875us) server_negotiation.cc:264] Negotiation successful 0506 07:57:47.025881 (+ 188us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":134} I0506 07:57:47.084882 13323 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:47.065347 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:36874 0506 07:57:47.065738 (+ 391us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:47.065763 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:47.072196 (+ 6433us) server_negotiation.cc:373] Connection header received 0506 07:57:47.072501 (+ 305us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:47.072527 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:47.072704 (+ 177us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:47.072822 (+ 118us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:47.076774 (+ 3952us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.077358 (+ 584us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.080130 (+ 2772us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.081590 (+ 1460us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.081792 (+ 202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:47.082980 (+ 1188us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:47.083192 (+ 212us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:47.083288 (+ 96us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:47.083513 (+ 225us) server_negotiation.cc:264] Negotiation successful 0506 07:57:47.083732 (+ 219us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":167} I0506 07:57:47.102843 13313 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:47.066337 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:56472 0506 07:57:47.066619 (+ 282us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:47.066639 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:47.084749 (+ 18110us) server_negotiation.cc:373] Connection header received 0506 07:57:47.085475 (+ 726us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:47.085508 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:47.085670 (+ 162us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:47.085848 (+ 178us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:47.089827 (+ 3979us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.090385 (+ 558us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.092934 (+ 2549us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.094364 (+ 1430us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.094546 (+ 182us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:47.100149 (+ 5603us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:47.100497 (+ 348us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:47.100606 (+ 109us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:47.101398 (+ 792us) server_negotiation.cc:264] Negotiation successful 0506 07:57:47.101577 (+ 179us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":78} I0506 07:57:47.108548 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 12248 I0506 07:57:47.116253 13321 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:47.090729 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:59166 0506 07:57:47.091021 (+ 292us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:47.091045 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:47.091084 (+ 39us) server_negotiation.cc:373] Connection header received 0506 07:57:47.094126 (+ 3042us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:47.094154 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:47.094334 (+ 180us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:47.094451 (+ 117us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:47.096179 (+ 1728us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.096827 (+ 648us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.102294 (+ 5467us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:47.103730 (+ 1436us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:47.103906 (+ 176us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:47.107262 (+ 3356us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:47.107481 (+ 219us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:47.107572 (+ 91us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:47.114842 (+ 7270us) server_negotiation.cc:264] Negotiation successful 0506 07:57:47.115019 (+ 177us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":86} W0506 07:57:47.164338 11759 connection.cc:512] server connection from 127.0.196.65:60320 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:60320): Connection reset by peer (error 104) W0506 07:57:47.165401 12049 connection.cc:512] server connection from 127.0.196.65:35781 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:35781): Connection reset by peer (error 104) I0506 07:57:47.166400 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 12627 W0506 07:57:47.166501 12187 connection.cc:512] server connection from 127.0.196.65:45756 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:45756): Connection reset by peer (error 104) W0506 07:57:47.239192 11759 connection.cc:512] server connection from 127.0.196.66:54781 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:54781): Connection reset by peer (error 104) W0506 07:57:47.239212 12152 connection.cc:512] server connection from 127.0.196.66:50795 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:50795): Connection reset by peer (error 104) W0506 07:57:47.240027 12007 connection.cc:512] server connection from 127.0.196.66:49224 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:49224): Connection reset by peer (error 104) I0506 07:57:47.241171 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 12932 W0506 07:57:47.276096 11759 connection.cc:512] server connection from 127.0.196.67:36106 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:36106): Connection reset by peer (error 104) W0506 07:57:47.278018 12008 connection.cc:512] server connection from 127.0.196.67:58626 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:58626): Connection reset by peer (error 104) W0506 07:57:47.279048 12149 connection.cc:512] server connection from 127.0.196.67:57952 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:57952): Connection reset by peer (error 104) I0506 07:57:47.280046 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 11750 I0506 07:57:47.348405 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 11865 I0506 07:57:47.413520 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 12113 [ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (10146 ms) [ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): setting up network... May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): skipping unrecognized local address family 17 May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): skipping unrecognized local address family 17 May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): set up 1 sockets May 06 07:57:47 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): commencing operation krb5kdc: starting... W0506 07:57:47.798041 785 mini_kdc.cc:117] Time spent starting KDC: real 0.259s user 0.000s sys 0.021s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. May 06 07:57:48 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.196.126@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.126@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. Entry for principal kudu/127.0.196.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab. I0506 07:57:48.070061 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:50795 --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:48.308992 13576 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:48.310114 13576 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:48.310326 13576 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.126 W0506 07:57:48.310770 13576 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:48.321657 13576 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:48.321888 13576 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:48.322072 13576 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:48.322250 13576 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:48.322453 13576 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:48.322695 13576 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.126 I0506 07:57:48.367720 13576 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.126:50795 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.126.keytab --principal=kudu/127.0.196.126 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.196.126 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:48.376962 13576 master_main.cc:73] Initializing master server... I0506 07:57:48.379062 13576 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 229678us May 06 07:57:48 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:48.386023 13576 init.cc:339] Logged in from keytab as kudu/127.0.196.126@KRBTEST.COM (short username kudu) I0506 07:57:48.403540 13576 fs_manager.cc:260] Metadata directory not provided I0506 07:57:48.403818 13576 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:48.404177 13576 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0506 07:57:48.404415 13576 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:48.419770 13576 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/instance: uuid: "d4d2993be16d445c8c7f502179e16cb5" format_stamp: "Formatted at 2018-05-06 14:57:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:48.420871 13576 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/wal/instance: uuid: "d4d2993be16d445c8c7f502179e16cb5" format_stamp: "Formatted at 2018-05-06 14:57:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:48.442257 13576 fs_manager.cc:495] Time spent creating directory manager: real 0.021s user 0.013s sys 0.004s I0506 07:57:48.443995 13576 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:48.445359 13576 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:48.463376 13576 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s I0506 07:57:48.463616 13576 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/wal uuid: "d4d2993be16d445c8c7f502179e16cb5" format_stamp: "Formatted at 2018-05-06 14:57:48 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:48.464033 13576 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:48.769502 13576 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:48.774425 13576 master_main.cc:76] Starting Master server... I0506 07:57:49.112154 13576 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.126:50795 I0506 07:57:49.113538 13576 webserver.cc:173] Starting webserver on 127.0.196.126:0 I0506 07:57:49.113749 13576 webserver.cc:184] Document root disabled I0506 07:57:49.129436 13576 webserver.cc:311] Webserver started. Bound to: http://127.0.196.126:57423/ I0506 07:57:49.131881 13576 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-0/data/info.pb I0506 07:57:49.139780 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 13576 I0506 07:57:49.145018 13900 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.125@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.125@KRBTEST.COM" created. I0506 07:57:49.240406 13900 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. Entry for principal kudu/127.0.196.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab. W0506 07:57:49.267652 785 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.196.125: real 0.127s user 0.000s sys 0.013s I0506 07:57:49.269269 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.125:45294 --webserver_interface=127.0.196.125 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab --principal=kudu/127.0.196.125 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:49 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:49.343281 13915 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.251238 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:52973 0506 07:57:49.254546 (+ 3308us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:49.254579 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:49.257232 (+ 2653us) server_negotiation.cc:373] Connection header received 0506 07:57:49.275235 (+ 18003us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:49.275266 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:49.275508 (+ 242us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:49.275763 (+ 255us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:49.278020 (+ 2257us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:49.278748 (+ 728us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:49.280757 (+ 2009us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:49.283060 (+ 2303us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:49.283275 (+ 215us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:49.300737 (+ 17462us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:49.300780 (+ 43us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:49.300810 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:49.300892 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:49.323487 (+ 22595us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:49.327070 (+ 3583us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:49.327100 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:49.327120 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:49.327818 (+ 698us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:49.336909 (+ 9091us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:49.336934 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:49.336952 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:49.337559 (+ 607us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:49.340840 (+ 3281us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:49.341070 (+ 230us) server_negotiation.cc:264] Negotiation successful 0506 07:57:49.341551 (+ 481us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3062,"thread_start_us":2221,"threads_started":1} I0506 07:57:49.344139 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.249794 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:49.257059 (+ 7265us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.257125 (+ 66us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:49.267243 (+ 10118us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:49.276149 (+ 8906us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:49.276178 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:49.277010 (+ 832us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:49.277268 (+ 258us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:49.277626 (+ 358us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:49.277678 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:49.279027 (+ 1349us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:49.279054 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:49.280440 (+ 1386us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:49.280484 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:49.283866 (+ 3382us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:49.283901 (+ 35us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:49.284300 (+ 399us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:49.285260 (+ 960us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:49.285284 (+ 24us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:49.300286 (+ 15002us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:49.325102 (+ 24816us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:49.325141 (+ 39us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:49.325169 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:49.326490 (+ 1321us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:49.328088 (+ 1598us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:49.328115 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:49.328136 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:49.328393 (+ 257us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:49.337764 (+ 9371us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:49.337792 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:49.338156 (+ 364us) client_negotiation.cc:695] Sending connection context 0506 07:57:49.338424 (+ 268us) client_negotiation.cc:225] Negotiation successful 0506 07:57:49.339070 (+ 646us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6981,"thread_start_us":9713,"threads_started":1} I0506 07:57:49.360628 13900 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:49.363435 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.362072 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:49.362536 (+ 464us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.363000 (+ 464us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":269} W0506 07:57:49.364202 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:49.383615 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 1 I0506 07:57:49.385596 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.384498 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:49.384994 (+ 496us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.385148 (+ 154us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":289} W0506 07:57:49.386070 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:49.434465 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 2 I0506 07:57:49.436395 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.435271 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:49.435788 (+ 517us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.435946 (+ 158us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":271} W0506 07:57:49.436974 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:49.548317 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 3 I0506 07:57:49.550649 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.549155 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:49.550082 (+ 927us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.550232 (+ 150us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":295} W0506 07:57:49.551244 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:49.727588 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 4 I0506 07:57:49.729413 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:49.728376 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:49.728862 (+ 486us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:49.729000 (+ 138us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":293} W0506 07:57:49.729984 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:49.976632 13917 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:49.977385 13917 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:49.977617 13917 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.125 W0506 07:57:49.978015 13917 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:49.996827 13917 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:49.997097 13917 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:49.997278 13917 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:49.997453 13917 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:49.997674 13917 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:49.997891 13917 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.125 I0506 07:57:50.033344 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 5 I0506 07:57:50.035295 13914 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.034189 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:50.034737 (+ 548us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.034879 (+ 142us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":336} W0506 07:57:50.035840 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:50.049159 13917 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.125:45294 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.125.keytab --principal=kudu/127.0.196.125 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/info.pb --webserver_interface=127.0.196.125 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:50.063400 13917 master_main.cc:73] Initializing master server... I0506 07:57:50.066864 13917 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 230678us May 06 07:57:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:50.076763 13917 init.cc:339] Logged in from keytab as kudu/127.0.196.125@KRBTEST.COM (short username kudu) I0506 07:57:50.095417 13917 fs_manager.cc:260] Metadata directory not provided I0506 07:57:50.095671 13917 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:50.096004 13917 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/instance: No such file or directory (error 2) I0506 07:57:50.096197 13917 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:50.111084 13917 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/instance: uuid: "8671bbcf68254398acccde133788f79a" format_stamp: "Formatted at 2018-05-06 14:57:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:50.112203 13917 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/wal/instance: uuid: "8671bbcf68254398acccde133788f79a" format_stamp: "Formatted at 2018-05-06 14:57:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:50.138502 13917 fs_manager.cc:495] Time spent creating directory manager: real 0.026s user 0.013s sys 0.006s I0506 07:57:50.141362 13917 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:50.142885 13917 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:50.167348 13917 fs_manager.cc:417] Time spent opening block manager: real 0.013s user 0.001s sys 0.009s I0506 07:57:50.167565 13917 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/wal uuid: "8671bbcf68254398acccde133788f79a" format_stamp: "Formatted at 2018-05-06 14:57:50 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:50.167963 13917 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:50.570192 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 6 I0506 07:57:50.583726 14014 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.570998 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:50.583043 (+ 12045us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.583201 (+ 158us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":11830,"thread_start_us":3857,"threads_started":1} W0506 07:57:50.584121 13900 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.125:45294: Network error: Client connection negotiation failed: client connection to 127.0.196.125:45294: connect: Connection refused (error 111) I0506 07:57:50.601593 13917 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:50.618610 13917 master_main.cc:76] Starting Master server... I0506 07:57:50.767616 13917 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.125:45294 I0506 07:57:50.769184 13917 webserver.cc:173] Starting webserver on 127.0.196.125:0 I0506 07:57:50.769366 13917 webserver.cc:184] Document root disabled I0506 07:57:50.775357 13917 webserver.cc:311] Webserver started. Bound to: http://127.0.196.125:48904/ I0506 07:57:50.776371 13917 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-1/data/info.pb I0506 07:57:50.778281 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 13917 I0506 07:57:50.785107 14073 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.124@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.124@KRBTEST.COM" created. I0506 07:57:50.803747 14073 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. Entry for principal kudu/127.0.196.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab. I0506 07:57:50.817842 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.124:43659 --webserver_interface=127.0.196.124 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab --principal=kudu/127.0.196.124 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:50.846230 14077 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.807879 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:52983 0506 07:57:50.810248 (+ 2369us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:50.810278 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:50.810317 (+ 39us) server_negotiation.cc:373] Connection header received 0506 07:57:50.815210 (+ 4893us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:50.815259 (+ 49us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:50.815377 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:50.815507 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:50.825979 (+ 10472us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.826445 (+ 466us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.828334 (+ 1889us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.829801 (+ 1467us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.829988 (+ 187us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:50.834971 (+ 4983us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:50.835001 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:50.835026 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:50.835107 (+ 81us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:50.840296 (+ 5189us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:50.841540 (+ 1244us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:50.841557 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:50.841568 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:50.842050 (+ 482us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:50.842595 (+ 545us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:50.842615 (+ 20us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:50.842629 (+ 14us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:50.842969 (+ 340us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:50.843131 (+ 162us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:50.843915 (+ 784us) server_negotiation.cc:264] Negotiation successful 0506 07:57:50.844348 (+ 433us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":856,"threads_started":1} I0506 07:57:50.847088 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.806617 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:50.809622 (+ 3005us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.809703 (+ 81us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:50.814904 (+ 5201us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:50.824006 (+ 9102us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:50.824040 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:50.824923 (+ 883us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:50.825236 (+ 313us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.825726 (+ 490us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:50.825780 (+ 54us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.826668 (+ 888us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.826717 (+ 49us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.828105 (+ 1388us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:50.828150 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.830051 (+ 1901us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.830083 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.830554 (+ 471us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:50.830914 (+ 360us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:50.830937 (+ 23us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:50.834524 (+ 3587us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:50.840447 (+ 5923us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:50.840468 (+ 21us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:50.840485 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:50.841400 (+ 915us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:50.842247 (+ 847us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:50.842264 (+ 17us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:50.842275 (+ 11us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:50.842443 (+ 168us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:50.843131 (+ 688us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:50.843159 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:50.843477 (+ 318us) client_negotiation.cc:695] Sending connection context 0506 07:57:50.843760 (+ 283us) client_negotiation.cc:225] Negotiation successful 0506 07:57:50.844352 (+ 592us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2379,"thread_start_us":1561,"threads_started":1} I0506 07:57:50.848744 14073 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } has no permanent_uuid. Determining permanent_uuid... May 06 07:57:50 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:50.869000 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.849853 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:50.850575 (+ 722us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.850607 (+ 32us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:50.850798 (+ 191us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:50.852382 (+ 1584us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:50.852420 (+ 38us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:50.852884 (+ 464us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:50.852981 (+ 97us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.853106 (+ 125us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:50.853136 (+ 30us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.853961 (+ 825us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.853978 (+ 17us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.854686 (+ 708us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:50.854733 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.856456 (+ 1723us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.856483 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:50.857025 (+ 542us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:50.857127 (+ 102us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:50.857138 (+ 11us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:50.858702 (+ 1564us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:50.863307 (+ 4605us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:50.863323 (+ 16us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:50.863334 (+ 11us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:50.864058 (+ 724us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:50.864868 (+ 810us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:50.864885 (+ 17us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:50.864896 (+ 11us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:50.864999 (+ 103us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:50.865739 (+ 740us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:50.865775 (+ 36us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:50.866061 (+ 286us) client_negotiation.cc:695] Sending connection context 0506 07:57:50.866220 (+ 159us) client_negotiation.cc:225] Negotiation successful 0506 07:57:50.866720 (+ 500us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":97} I0506 07:57:50.870160 14080 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.849949 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47331 0506 07:57:50.851859 (+ 1910us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:50.851877 (+ 18us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:50.851908 (+ 31us) server_negotiation.cc:373] Connection header received 0506 07:57:50.851990 (+ 82us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:50.852009 (+ 19us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:50.852132 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:50.852266 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:50.853317 (+ 1051us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.853787 (+ 470us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.854888 (+ 1101us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:50.856268 (+ 1380us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:50.856434 (+ 166us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:50.858969 (+ 2535us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:50.858997 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:50.859013 (+ 16us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:50.859060 (+ 47us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:50.863162 (+ 4102us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:50.864201 (+ 1039us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:50.864221 (+ 20us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:50.864231 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:50.864722 (+ 491us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:50.865130 (+ 408us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:50.865147 (+ 17us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:50.865158 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:50.865572 (+ 414us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:50.865933 (+ 361us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:50.866419 (+ 486us) server_negotiation.cc:264] Negotiation successful 0506 07:57:50.867072 (+ 653us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":204,"server-negotiator.queue_time_us":1720,"thread_start_us":1315,"threads_started":1} I0506 07:57:50.871960 14073 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:50.873240 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.872537 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:50.872814 (+ 277us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.872976 (+ 162us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":93} W0506 07:57:50.873572 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) I0506 07:57:50.892995 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 1 I0506 07:57:50.894500 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.893766 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:50.894101 (+ 335us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.894221 (+ 120us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":114} W0506 07:57:50.894917 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) I0506 07:57:50.943279 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 2 I0506 07:57:50.944954 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:50.944043 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:50.944319 (+ 276us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:50.944472 (+ 153us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":73} W0506 07:57:50.945400 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:51.014947 14078 flags.cc:412] Enabled unsafe flag: --enable_leader_failure_detection=false W0506 07:57:51.015524 14078 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:51.015702 14078 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.124 W0506 07:57:51.015975 14078 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:51.027389 14078 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0506 07:57:51.027549 14078 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:51.027685 14078 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:51.027830 14078 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0506 07:57:51.027972 14078 flags.cc:412] Enabled experimental flag: --rpc_reuseport=true W0506 07:57:51.028121 14078 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.124 I0506 07:57:51.056725 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 3 I0506 07:57:51.058383 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.057497 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:51.057810 (+ 313us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.057960 (+ 150us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0506 07:57:51.058837 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) I0506 07:57:51.072762 14078 master_main.cc:66] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.196.124:43659 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.124.keytab --principal=kudu/127.0.196.124 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/info.pb --webserver_interface=127.0.196.124 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:51.090229 14078 master_main.cc:73] Initializing master server... I0506 07:57:51.092531 14078 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 231178us May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:51.095019 14078 init.cc:339] Logged in from keytab as kudu/127.0.196.124@KRBTEST.COM (short username kudu) I0506 07:57:51.098080 14078 fs_manager.cc:260] Metadata directory not provided I0506 07:57:51.098214 14078 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:51.098434 14078 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/instance: No such file or directory (error 2) I0506 07:57:51.098567 14078 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:51.115404 14078 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/instance: uuid: "731d40aa19f0477cbf46586776702a61" format_stamp: "Formatted at 2018-05-06 14:57:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:51.120774 14078 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/wal/instance: uuid: "731d40aa19f0477cbf46586776702a61" format_stamp: "Formatted at 2018-05-06 14:57:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:51.139576 14078 fs_manager.cc:495] Time spent creating directory manager: real 0.018s user 0.011s sys 0.004s I0506 07:57:51.141541 14078 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:51.142354 14078 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:51.159124 14078 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.003s sys 0.002s I0506 07:57:51.159324 14078 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/wal uuid: "731d40aa19f0477cbf46586776702a61" format_stamp: "Formatted at 2018-05-06 14:57:51 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:51.159734 14078 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:51.244432 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 4 I0506 07:57:51.246182 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.245286 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:51.245574 (+ 288us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.245753 (+ 179us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":85} W0506 07:57:51.246713 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) I0506 07:57:51.434882 14078 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:51.442760 14078 master_main.cc:76] Starting Master server... I0506 07:57:51.550065 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 5 I0506 07:57:51.551697 14076 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.550852 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:51.551146 (+ 294us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.551292 (+ 146us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":90} W0506 07:57:51.552106 14073 consensus_peers.cc:563] Error getting permanent uuid from config peer 127.0.196.124:43659: Network error: Client connection negotiation failed: client connection to 127.0.196.124:43659: connect: Connection refused (error 111) I0506 07:57:51.650640 14078 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.124:43659 I0506 07:57:51.652315 14078 webserver.cc:173] Starting webserver on 127.0.196.124:0 I0506 07:57:51.652503 14078 webserver.cc:184] Document root disabled I0506 07:57:51.658489 14078 webserver.cc:311] Webserver started. Bound to: http://127.0.196.124:36035/ I0506 07:57:51.660109 14078 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/master-2/data/info.pb I0506 07:57:51.664741 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 14078 I0506 07:57:51.666409 14281 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.0.196.65@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.65@KRBTEST.COM" created. I0506 07:57:51.696552 14281 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:51.624493 13900 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } attempt: 7 May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:51.763048 14289 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.701121 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:51.715467 (+ 14346us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.715545 (+ 78us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:51.721777 (+ 6232us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:51.723741 (+ 1964us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:51.724047 (+ 306us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:51.724955 (+ 908us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:51.725586 (+ 631us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.726073 (+ 487us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.726129 (+ 56us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.727470 (+ 1341us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.727494 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.729359 (+ 1865us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.729410 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.731982 (+ 2572us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.732013 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.732487 (+ 474us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.732781 (+ 294us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:51.732803 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:51.738307 (+ 5504us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:51.753908 (+ 15601us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.753935 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.753958 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.755240 (+ 1282us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.757326 (+ 2086us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.757352 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.757369 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.757659 (+ 290us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.758696 (+ 1037us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:51.758723 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:51.759159 (+ 436us) client_negotiation.cc:695] Sending connection context 0506 07:57:51.759488 (+ 329us) client_negotiation.cc:225] Negotiation successful 0506 07:57:51.760106 (+ 618us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13824,"spinlock_wait_cycles":361728,"thread_start_us":13360,"threads_started":1} I0506 07:57:51.769551 14288 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.701253 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:52995 0506 07:57:51.715795 (+ 14542us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:51.715823 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:51.715863 (+ 40us) server_negotiation.cc:373] Connection header received 0506 07:57:51.722944 (+ 7081us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:51.722971 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:51.723092 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:51.723265 (+ 173us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:51.726367 (+ 3102us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.727163 (+ 796us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.729670 (+ 2507us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.731428 (+ 1758us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.731630 (+ 202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.739920 (+ 8290us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:51.739952 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:51.739976 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:51.740049 (+ 73us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:51.753717 (+ 13668us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.755468 (+ 1751us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.755493 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.755511 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.756120 (+ 609us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.757880 (+ 1760us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.757905 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.757924 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.758375 (+ 451us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:51.758538 (+ 163us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:51.764987 (+ 6449us) server_negotiation.cc:264] Negotiation successful 0506 07:57:51.765559 (+ 572us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14309,"thread_start_us":14572,"threads_started":1} I0506 07:57:51.771517 14281 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:51.798930 14292 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.717076 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:51.734886 (+ 17810us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.734932 (+ 46us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:51.735241 (+ 309us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:51.736169 (+ 928us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:51.736195 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:51.736882 (+ 687us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:51.737050 (+ 168us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.737258 (+ 208us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.737303 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.738296 (+ 993us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.738323 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.739541 (+ 1218us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.739583 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.742120 (+ 2537us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.742147 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.742548 (+ 401us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.742747 (+ 199us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:51.742769 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:51.750789 (+ 8020us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:51.779748 (+ 28959us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.779778 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.779801 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.784959 (+ 5158us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.788848 (+ 3889us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.788874 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.788894 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.789052 (+ 158us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.790180 (+ 1128us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:51.790207 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:51.791679 (+ 1472us) client_negotiation.cc:695] Sending connection context 0506 07:57:51.791900 (+ 221us) client_negotiation.cc:225] Negotiation successful 0506 07:57:51.792413 (+ 513us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":17570,"mutex_wait_us":5089,"thread_start_us":17205,"threads_started":1} Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. Entry for principal kudu/127.0.196.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab. W0506 07:57:51.802234 785 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.196.65: real 0.137s user 0.001s sys 0.011s I0506 07:57:51.802752 14291 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.716782 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47343 0506 07:57:51.722788 (+ 6006us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:51.722813 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:51.735003 (+ 12190us) server_negotiation.cc:373] Connection header received 0506 07:57:51.735483 (+ 480us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:51.735510 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:51.735631 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:51.735809 (+ 178us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:51.737499 (+ 1690us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.738046 (+ 547us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.739810 (+ 1764us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.741361 (+ 1551us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.741542 (+ 181us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.751781 (+ 10239us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:51.751811 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:51.751834 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:51.751906 (+ 72us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:51.779411 (+ 27505us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.785325 (+ 5914us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.785353 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.785372 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.786127 (+ 755us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.789284 (+ 3157us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.789311 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.789328 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.789870 (+ 542us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:51.790050 (+ 180us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:51.793871 (+ 3821us) server_negotiation.cc:264] Negotiation successful 0506 07:57:51.794317 (+ 446us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5734,"thread_start_us":5882,"threads_started":1} I0506 07:57:51.803807 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.0.196.65:0 --local_ip_for_outbound_sockets=127.0.196.65 --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0506 07:57:51.805428 13900 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid... May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.126@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:51.870483 14303 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.773089 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47346 0506 07:57:51.790611 (+ 17522us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:51.790663 (+ 52us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:51.790702 (+ 39us) server_negotiation.cc:373] Connection header received 0506 07:57:51.790800 (+ 98us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:51.790824 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:51.790939 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:51.791096 (+ 157us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:51.792394 (+ 1298us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.792970 (+ 576us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.794826 (+ 1856us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.808076 (+ 13250us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.812225 (+ 4149us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.812540 (+ 315us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:51.812567 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:51.812591 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:51.812691 (+ 100us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:51.826811 (+ 14120us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.860015 (+ 33204us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.860045 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.860064 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.860776 (+ 712us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.863532 (+ 2756us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.863557 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.863576 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.864101 (+ 525us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:51.864287 (+ 186us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:51.868033 (+ 3746us) server_negotiation.cc:264] Negotiation successful 0506 07:57:51.868586 (+ 553us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4165,"thread_start_us":3861,"threads_started":1} I0506 07:57:51.871290 14289 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.772407 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:51.772745 (+ 338us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.772784 (+ 39us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:51.772921 (+ 137us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:51.791165 (+ 18244us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:51.791190 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:51.791826 (+ 636us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:51.791978 (+ 152us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.792174 (+ 196us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.792213 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.793298 (+ 1085us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.793324 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.794611 (+ 1287us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.794657 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.808728 (+ 14071us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.808757 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.809161 (+ 404us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.809322 (+ 161us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:51.809342 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:51.811943 (+ 2601us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:51.836661 (+ 24718us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.836686 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.836707 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.859781 (+ 23074us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.863123 (+ 3342us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.863150 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.863170 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.863337 (+ 167us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.864612 (+ 1275us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:51.864638 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:51.864989 (+ 351us) client_negotiation.cc:695] Sending connection context 0506 07:57:51.865214 (+ 225us) client_negotiation.cc:225] Negotiation successful 0506 07:57:51.865793 (+ 579us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":21834} I0506 07:57:51.873160 14281 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } has no permanent_uuid. Determining permanent_uuid... I0506 07:57:51.917685 14292 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.806191 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:51.806846 (+ 655us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.806880 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:51.807004 (+ 124us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:51.816962 (+ 9958us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:51.816987 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:51.817610 (+ 623us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:51.817785 (+ 175us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.817981 (+ 196us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.818024 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.819733 (+ 1709us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.819759 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.820974 (+ 1215us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.821019 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.826950 (+ 5931us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.826978 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.827371 (+ 393us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.827539 (+ 168us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:51.827558 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:51.830190 (+ 2632us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:51.905906 (+ 75716us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.905936 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.905957 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.907309 (+ 1352us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.911735 (+ 4426us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.911761 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.911782 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.911946 (+ 164us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.914577 (+ 2631us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:51.914604 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:51.914957 (+ 353us) client_negotiation.cc:695] Sending connection context 0506 07:57:51.915166 (+ 209us) client_negotiation.cc:225] Negotiation successful 0506 07:57:51.915743 (+ 577us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":433} I0506 07:57:51.933014 14308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.806860 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38405 0506 07:57:51.816202 (+ 9342us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:51.816230 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:51.816273 (+ 43us) server_negotiation.cc:373] Connection header received 0506 07:57:51.816373 (+ 100us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:51.816399 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:51.816565 (+ 166us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:51.818185 (+ 1620us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:51.818515 (+ 330us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.819276 (+ 761us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.821310 (+ 2034us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.826638 (+ 5328us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.829008 (+ 2370us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.832491 (+ 3483us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:51.832533 (+ 42us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:51.832563 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:51.832637 (+ 74us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:51.899246 (+ 66609us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.910439 (+ 11193us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.910471 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.910491 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.911271 (+ 780us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.912202 (+ 931us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.912229 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.912250 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.914359 (+ 2109us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:51.921464 (+ 7105us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:51.923205 (+ 1741us) server_negotiation.cc:264] Negotiation successful 0506 07:57:51.925193 (+ 1988us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9135,"thread_start_us":6169,"threads_started":1} May 06 07:57:51 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.124@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:51.941648 13900 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: Bootstrap starting. I0506 07:57:51.951032 13900 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: No blocks or log segments found. Creating new log. I0506 07:57:51.952808 13900 log.cc:520] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: Log is configured to *not* fsync() on all Append() calls I0506 07:57:51.981230 14289 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.873995 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:51.891665 (+ 17670us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:51.891727 (+ 62us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:51.891865 (+ 138us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:51.912548 (+ 20683us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:51.912576 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:51.913268 (+ 692us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:51.913429 (+ 161us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.913644 (+ 215us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.913712 (+ 68us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.925830 (+ 12118us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.925859 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.927187 (+ 1328us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:51.927234 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.938652 (+ 11418us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.938702 (+ 50us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:51.939117 (+ 415us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.939282 (+ 165us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:51.939302 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:51.942246 (+ 2944us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:51.973650 (+ 31404us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.973702 (+ 52us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.973725 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.974996 (+ 1271us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.976961 (+ 1965us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:51.976989 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:51.977009 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:51.977156 (+ 147us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:51.978097 (+ 941us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:51.978122 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:51.978434 (+ 312us) client_negotiation.cc:695] Sending connection context 0506 07:57:51.978689 (+ 255us) client_negotiation.cc:225] Negotiation successful 0506 07:57:51.979237 (+ 548us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":17471,"thread_start_us":37318,"threads_started":1} I0506 07:57:51.988512 14315 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:51.875057 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38407 0506 07:57:51.911986 (+ 36929us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:51.912013 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:51.912050 (+ 37us) server_negotiation.cc:373] Connection header received 0506 07:57:51.912146 (+ 96us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:51.912170 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:51.912286 (+ 116us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:51.912470 (+ 184us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:51.918746 (+ 6276us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.924848 (+ 6102us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.927542 (+ 2694us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:51.936909 (+ 9367us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:51.947519 (+ 10610us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:51.947902 (+ 383us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:51.947932 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:51.947962 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:51.948037 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:51.970217 (+ 22180us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.976071 (+ 5854us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.976099 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.976117 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.976802 (+ 685us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:51.977350 (+ 548us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:51.977373 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:51.977392 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:51.977913 (+ 521us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:51.982648 (+ 4735us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:51.982898 (+ 250us) server_negotiation.cc:264] Negotiation successful 0506 07:57:51.983740 (+ 842us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":36725,"thread_start_us":26505,"threads_started":1} I0506 07:57:51.993629 14281 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: Bootstrap starting. I0506 07:57:51.997691 14281 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: No blocks or log segments found. Creating new log. I0506 07:57:51.997728 13900 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: No bootstrap required, opened a new log I0506 07:57:51.999198 14281 log.cc:520] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: Log is configured to *not* fsync() on all Append() calls I0506 07:57:52.034641 14281 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: No bootstrap required, opened a new log I0506 07:57:52.033408 13900 raft_consensus.cc:303] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.036154 13900 raft_consensus.cc:604] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d2993be16d445c8c7f502179e16cb5, State: Initialized, Role: FOLLOWER I0506 07:57:52.036711 13900 consensus_queue.cc:227] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.043145 14325 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.043892 14325 sys_catalog.cc:338] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:52.044342 13900 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.044984 13900 sys_catalog.cc:338] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:52.046334 13900 sys_catalog.cc:422] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:52.047919 14326 catalog_manager.cc:504] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: 1 I0506 07:57:52.049744 14281 raft_consensus.cc:303] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.050253 14281 raft_consensus.cc:604] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731d40aa19f0477cbf46586776702a61, State: Initialized, Role: FOLLOWER I0506 07:57:52.050618 14281 consensus_queue.cc:227] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.052456 14281 sys_catalog.cc:335] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.053256 14281 sys_catalog.cc:338] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:52.054517 14281 sys_catalog.cc:422] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:52.057448 13576 master_main.cc:79] Master server successfully started. I0506 07:57:52.052845 14327 sys_catalog.cc:335] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.118083 14327 sys_catalog.cc:338] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:52.124585 14078 master_main.cc:79] Master server successfully started. W0506 07:57:52.135859 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:52.136123 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:52.196182 14185 tablet.cc:1678] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:52.086467 14073 consensus_peers.cc:573] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } attempt: 6 W0506 07:57:52.170390 14016 debug-util.cc:397] Leaking SignalData structure 0x7b080002a060 after lost signal to thread 13987 WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:52.247776 14309 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:52.248240 14309 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.65 W0506 07:57:52.248734 14309 flags.cc:412] Enabled unsafe flag: --never_fsync=true May 06 07:57:52 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.125@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM W0506 07:57:52.266542 14309 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:52.266782 14309 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:52.267151 14309 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.65 W0506 07:57:52.267735 14309 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.65 I0506 07:57:52.275529 14315 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:52.230663 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38410 0506 07:57:52.230991 (+ 328us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:52.231011 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:52.251871 (+ 20860us) server_negotiation.cc:373] Connection header received 0506 07:57:52.252119 (+ 248us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:52.252148 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:52.252282 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:52.252458 (+ 176us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:52.253995 (+ 1537us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:52.254489 (+ 494us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:52.256169 (+ 1680us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:52.257650 (+ 1481us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:52.257869 (+ 219us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:52.261938 (+ 4069us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:52.261967 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:52.261991 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:52.262068 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:52.268599 (+ 6531us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:52.270196 (+ 1597us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:52.270225 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:52.270244 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:52.271201 (+ 957us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:52.271860 (+ 659us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:52.271887 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:52.271907 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:52.272409 (+ 502us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:52.272603 (+ 194us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:52.273300 (+ 697us) server_negotiation.cc:264] Negotiation successful 0506 07:57:52.273754 (+ 454us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":94} I0506 07:57:52.275619 14342 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:52.230123 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:52.251728 (+ 21605us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:52.251775 (+ 47us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:52.251921 (+ 146us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:52.252497 (+ 576us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:52.252523 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:52.253273 (+ 750us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:52.253532 (+ 259us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:52.253780 (+ 248us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:52.253824 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:52.254679 (+ 855us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:52.254707 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:52.255950 (+ 1243us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:52.255992 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:52.257890 (+ 1898us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:52.257917 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:52.258382 (+ 465us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:52.258551 (+ 169us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:52.258569 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:52.261689 (+ 3120us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:52.268792 (+ 7103us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:52.268817 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:52.268836 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:52.270033 (+ 1197us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:52.271472 (+ 1439us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:52.271498 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:52.271517 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:52.271698 (+ 181us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:52.272566 (+ 868us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:52.272590 (+ 24us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:52.272912 (+ 322us) client_negotiation.cc:695] Sending connection context 0506 07:57:52.273116 (+ 204us) client_negotiation.cc:225] Negotiation successful 0506 07:57:52.273612 (+ 496us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15984,"thread_start_us":16349,"threads_started":1} I0506 07:57:52.282546 14073 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: Bootstrap starting. W0506 07:57:52.286237 13783 tablet.cc:1678] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:52.286386 14073 tablet_bootstrap.cc:587] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: No blocks or log segments found. Creating new log. I0506 07:57:52.288163 14073 log.cc:520] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: Log is configured to *not* fsync() on all Append() calls I0506 07:57:52.303618 14073 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: No bootstrap required, opened a new log I0506 07:57:52.321403 14309 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.65:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.65.keytab --principal=kudu/127.0.196.65 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.196.65 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:52.345559 14309 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:52.353772 14309 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 231678us I0506 07:57:52.354503 14073 raft_consensus.cc:303] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.355360 14073 raft_consensus.cc:604] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8671bbcf68254398acccde133788f79a, State: Initialized, Role: FOLLOWER I0506 07:57:52.355916 14073 consensus_queue.cc:227] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:52.362771 14356 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.363492 14073 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:52.364173 14073 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:52.365485 14073 sys_catalog.cc:422] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: configured and running, proceeding with master startup. I0506 07:57:52.366437 14356 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: This master's current role is: FOLLOWER May 06 07:57:52 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:52.372328 14309 init.cc:339] Logged in from keytab as kudu/127.0.196.65@KRBTEST.COM (short username kudu) I0506 07:57:52.377401 13917 master_main.cc:79] Master server successfully started. W0506 07:57:52.379266 14025 tablet.cc:1678] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0506 07:57:52.393322 14309 fs_manager.cc:260] Metadata directory not provided I0506 07:57:52.393553 14309 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:52.393923 14309 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/instance: No such file or directory (error 2) W0506 07:57:52.393996 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:52.394203 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:52.394224 14309 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:52.414441 14309 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/instance: uuid: "67bbccebf4ed46a594886a8a20235ec9" format_stamp: "Formatted at 2018-05-06 14:57:52 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:52.429036 14309 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/wal/instance: uuid: "67bbccebf4ed46a594886a8a20235ec9" format_stamp: "Formatted at 2018-05-06 14:57:52 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:52.484747 14309 fs_manager.cc:495] Time spent creating directory manager: real 0.054s user 0.013s sys 0.003s I0506 07:57:52.486851 14309 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:52.487488 14309 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:52.526608 14309 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s I0506 07:57:52.526965 14309 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/wal uuid: "67bbccebf4ed46a594886a8a20235ec9" format_stamp: "Formatted at 2018-05-06 14:57:52 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:52.527390 14309 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:52.837126 14309 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:52.853878 14309 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:52.866936 14309 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:53.056309 14326 catalog_manager.cc:1036] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:53.056557 14326 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:53.138648 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:53.138856 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:53.396572 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:53.396803 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:53.708832 14309 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.65:41770 I0506 07:57:53.710500 14309 webserver.cc:173] Starting webserver on 127.0.196.65:0 I0506 07:57:53.710670 14309 webserver.cc:184] Document root disabled I0506 07:57:53.721262 14309 webserver.cc:311] Webserver started. Bound to: http://127.0.196.65:33624/ I0506 07:57:53.722447 14309 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-0/data/info.pb I0506 07:57:53.725646 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 14309 WARNING: no policy specified for kudu/127.0.196.66@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.66@KRBTEST.COM" created. I0506 07:57:53.796288 14309 tablet_server_main.cc:87] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. Entry for principal kudu/127.0.196.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab. I0506 07:57:53.810269 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.0.196.66:0 --local_ip_for_outbound_sockets=127.0.196.66 --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:53 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:53 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:53 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.65@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:53.865614 14540 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.733649 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:34568 0506 07:57:53.753403 (+ 19754us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:53.753428 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:53.753469 (+ 41us) server_negotiation.cc:373] Connection header received 0506 07:57:53.792991 (+ 39522us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:53.793019 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:53.793136 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:53.793286 (+ 150us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:53.801411 (+ 8125us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.801980 (+ 569us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.819278 (+ 17298us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.820827 (+ 1549us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.821003 (+ 176us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.830238 (+ 9235us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:53.830320 (+ 82us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:53.830350 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:53.830434 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:53.837572 (+ 7138us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.841490 (+ 3918us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.841517 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.841536 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.842202 (+ 666us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.847003 (+ 4801us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.847027 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.847045 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.847654 (+ 609us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:53.857112 (+ 9458us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:53.863387 (+ 6275us) server_negotiation.cc:264] Negotiation successful 0506 07:57:53.863763 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19477,"thread_start_us":19170,"threads_started":1} I0506 07:57:53.867452 14541 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.733976 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:53.741950 (+ 7974us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:53.742094 (+ 144us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:53.791033 (+ 48939us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:53.794447 (+ 3414us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:53.794475 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:53.800571 (+ 6096us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:53.800785 (+ 214us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.801172 (+ 387us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.801213 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.802774 (+ 1561us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.802801 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.818921 (+ 16120us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.818982 (+ 61us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.821449 (+ 2467us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.821480 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.821932 (+ 452us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.822146 (+ 214us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:53.822165 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:53.829446 (+ 7281us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:53.838287 (+ 8841us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.838318 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.838343 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.840812 (+ 2469us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.842885 (+ 2073us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.842914 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.842933 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.843191 (+ 258us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.854980 (+ 11789us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:53.855011 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:53.855507 (+ 496us) client_negotiation.cc:695] Sending connection context 0506 07:57:53.863140 (+ 7633us) client_negotiation.cc:225] Negotiation successful 0506 07:57:53.865084 (+ 1944us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7386,"mutex_wait_us":26740,"thread_start_us":6885,"threads_started":1} I0506 07:57:53.879827 14537 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50795 I0506 07:57:53.880144 14537 heartbeater.cc:426] Registering TS with master... I0506 07:57:53.882199 14537 heartbeater.cc:475] Master 127.0.196.126:50795 requested a full tablet report, sending... I0506 07:57:53.876121 14542 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.748743 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:58455 0506 07:57:53.763049 (+ 14306us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:53.763078 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:53.769181 (+ 6103us) server_negotiation.cc:373] Connection header received 0506 07:57:53.793767 (+ 24586us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:53.793796 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:53.793921 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:53.794064 (+ 143us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:53.797478 (+ 3414us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.798054 (+ 576us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.807340 (+ 9286us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.808858 (+ 1518us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.809035 (+ 177us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.828560 (+ 19525us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:53.828641 (+ 81us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:53.828696 (+ 55us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:53.828780 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:53.835902 (+ 7122us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.846973 (+ 11071us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.847001 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.847025 (+ 24us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.847792 (+ 767us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.859486 (+ 11694us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.859513 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.859530 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.860058 (+ 528us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:53.860444 (+ 386us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:53.871066 (+ 10622us) server_negotiation.cc:264] Negotiation successful 0506 07:57:53.874326 (+ 3260us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14027,"thread_start_us":8887,"threads_started":1} I0506 07:57:53.885751 13819 ts_manager.cc:81] Registered new tserver with Master: 67bbccebf4ed46a594886a8a20235ec9 (127.0.196.65:41770) I0506 07:57:53.887845 14543 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.748841 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:53.768939 (+ 20098us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:53.769085 (+ 146us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:53.792411 (+ 23326us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:53.794447 (+ 2036us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:53.794474 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:53.796170 (+ 1696us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:53.796692 (+ 522us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.797195 (+ 503us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.797245 (+ 50us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.799848 (+ 2603us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.799876 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.806977 (+ 7101us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.807066 (+ 89us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.813797 (+ 6731us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.813828 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.818355 (+ 4527us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.818865 (+ 510us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:53.818891 (+ 26us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:53.827127 (+ 8236us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:53.845078 (+ 17951us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.845105 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.845126 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.846320 (+ 1194us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.857095 (+ 10775us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.857121 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.857141 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.858099 (+ 958us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.863406 (+ 5307us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:53.863440 (+ 34us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:53.868522 (+ 5082us) client_negotiation.cc:695] Sending connection context 0506 07:57:53.868971 (+ 449us) client_negotiation.cc:225] Negotiation successful 0506 07:57:53.869285 (+ 314us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":19865,"mutex_wait_us":13038,"thread_start_us":9310,"threads_started":1} I0506 07:57:53.890260 14547 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.770924 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.65:35126 0506 07:57:53.776629 (+ 5705us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:53.776677 (+ 48us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:53.776715 (+ 38us) server_negotiation.cc:373] Connection header received 0506 07:57:53.791924 (+ 15209us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:53.791950 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:53.792065 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:53.792200 (+ 135us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:53.800363 (+ 8163us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.800951 (+ 588us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.821689 (+ 20738us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.823242 (+ 1553us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.823464 (+ 222us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.848497 (+ 25033us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:53.848563 (+ 66us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:53.848585 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:53.848700 (+ 115us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:53.855319 (+ 6619us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.858178 (+ 2859us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.858203 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.858218 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.858851 (+ 633us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:53.861727 (+ 2876us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:53.861753 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:53.861772 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:53.862256 (+ 484us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:53.862642 (+ 386us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:53.887557 (+ 24915us) server_negotiation.cc:264] Negotiation successful 0506 07:57:53.887930 (+ 373us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5502,"thread_start_us":5225,"threads_started":1} I0506 07:57:53.891348 14538 heartbeater.cc:346] Connected to a master server at 127.0.196.125:45294 I0506 07:57:53.891584 14538 heartbeater.cc:426] Registering TS with master... I0506 07:57:53.892092 14538 heartbeater.cc:475] Master 127.0.196.125:45294 requested a full tablet report, sending... I0506 07:57:53.898571 14039 ts_manager.cc:81] Registered new tserver with Master: 67bbccebf4ed46a594886a8a20235ec9 (127.0.196.65:41770) I0506 07:57:53.892910 14546 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:53.770738 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:53.775564 (+ 4826us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:53.775770 (+ 206us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:53.791698 (+ 15928us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:53.793249 (+ 1551us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:53.793286 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:53.798895 (+ 5609us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:53.799191 (+ 296us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.799616 (+ 425us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.799684 (+ 68us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.801215 (+ 1531us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.801239 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.820058 (+ 18819us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:53.820103 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:53.824029 (+ 3926us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:53.824057 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:53.824466 (+ 409us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:53.824753 (+ 287us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:53.824772 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:53.839240 (+ 14468us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:53.856250 (+ 17010us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.856274 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.856290 (+ 16us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.857488 (+ 1198us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.860851 (+ 3363us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:53.860878 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:53.860897 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:53.861058 (+ 161us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:53.864591 (+ 3533us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:53.864618 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:53.884210 (+ 19592us) client_negotiation.cc:695] Sending connection context 0506 07:57:53.884628 (+ 418us) client_negotiation.cc:225] Negotiation successful 0506 07:57:53.884961 (+ 333us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":4567,"mutex_wait_us":41241,"thread_start_us":4264,"threads_started":1} I0506 07:57:53.975139 14544 heartbeater.cc:346] Connected to a master server at 127.0.196.124:43659 I0506 07:57:53.975392 14544 heartbeater.cc:426] Registering TS with master... I0506 07:57:53.975913 14544 heartbeater.cc:475] Master 127.0.196.124:43659 requested a full tablet report, sending... I0506 07:57:53.979203 14201 ts_manager.cc:81] Registered new tserver with Master: 67bbccebf4ed46a594886a8a20235ec9 (127.0.196.65:41770) W0506 07:57:54.058990 14326 catalog_manager.cc:1036] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:54.059202 14326 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:54.141306 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:54.141520 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:54.233960 14549 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:54.234522 14549 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.66 W0506 07:57:54.235002 14549 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:54.255993 14549 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:54.256238 14549 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:54.256541 14549 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.66 W0506 07:57:54.257149 14549 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.66 I0506 07:57:54.291357 14549 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.66:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.66.keytab --principal=kudu/127.0.196.66 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.196.66 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:54.330332 14549 tablet_server_main.cc:79] Initializing tablet server... I0506 07:57:54.338542 14549 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 232678us May 06 07:57:54 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:54.345453 14549 init.cc:339] Logged in from keytab as kudu/127.0.196.66@KRBTEST.COM (short username kudu) I0506 07:57:54.361889 14549 fs_manager.cc:260] Metadata directory not provided I0506 07:57:54.362114 14549 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:54.362465 14549 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0506 07:57:54.362758 14549 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:54.392952 14549 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/instance: uuid: "9e5d664019fd4fecb8189e7e5afc0f8a" format_stamp: "Formatted at 2018-05-06 14:57:54 on quasar-rkgotj-1.vpc.cloudera.com" W0506 07:57:54.399163 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:54.399370 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:54.404222 14549 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/wal/instance: uuid: "9e5d664019fd4fecb8189e7e5afc0f8a" format_stamp: "Formatted at 2018-05-06 14:57:54 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:54.485821 14549 fs_manager.cc:495] Time spent creating directory manager: real 0.081s user 0.013s sys 0.003s I0506 07:57:54.487854 14549 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:54.488549 14549 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:54.517889 14549 fs_manager.cc:417] Time spent opening block manager: real 0.018s user 0.004s sys 0.010s I0506 07:57:54.518147 14549 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/wal uuid: "9e5d664019fd4fecb8189e7e5afc0f8a" format_stamp: "Formatted at 2018-05-06 14:57:54 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:54.518704 14549 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:54.817888 14549 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:54.827133 14549 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0506 07:57:54.832733 14549 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:55.062759 14326 catalog_manager.cc:1036] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:55.062973 14326 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:55.147002 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:55.147210 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:55.401813 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:55.402029 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:55.418495 14382 debug-util.cc:397] Leaking SignalData structure 0x7b080002a080 after lost signal to thread 14348 W0506 07:57:55.419397 14382 debug-util.cc:397] Leaking SignalData structure 0x7b08000a6b40 after lost signal to thread 14535 I0506 07:57:55.903820 14549 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.66:41644 I0506 07:57:55.907359 14549 webserver.cc:173] Starting webserver on 127.0.196.66:0 I0506 07:57:55.907570 14549 webserver.cc:184] Document root disabled I0506 07:57:55.915920 14549 webserver.cc:311] Webserver started. Bound to: http://127.0.196.66:54442/ I0506 07:57:55.917058 14549 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-1/data/info.pb I0506 07:57:55.922062 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 14549 WARNING: no policy specified for kudu/127.0.196.67@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.196.67@KRBTEST.COM" created. I0506 07:57:56.008281 14549 tablet_server_main.cc:87] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. Entry for principal kudu/127.0.196.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab. I0506 07:57:56.018147 785 external_mini_cluster.cc:786] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.0.196.67:0 --local_ip_for_outbound_sockets=127.0.196.67 --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} May 06 07:57:56 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM May 06 07:57:56 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM May 06 07:57:56 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.66@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:56.063778 14692 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:55.933557 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:53568 0506 07:57:55.948911 (+ 15354us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:55.948939 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:55.948980 (+ 41us) server_negotiation.cc:373] Connection header received 0506 07:57:56.008657 (+ 59677us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:56.008687 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:56.008807 (+ 120us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:56.008967 (+ 160us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:56.012560 (+ 3593us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.013550 (+ 990us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.024903 (+ 11353us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.026407 (+ 1504us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.026625 (+ 218us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.033257 (+ 6632us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:56.033330 (+ 73us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:56.033355 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:56.033431 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:56.040310 (+ 6879us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.053402 (+ 13092us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.053426 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.053442 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.054068 (+ 626us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.057807 (+ 3739us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.057833 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.057850 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.058340 (+ 490us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:56.058745 (+ 405us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:56.061639 (+ 2894us) server_negotiation.cc:264] Negotiation successful 0506 07:57:56.061952 (+ 313us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15141,"thread_start_us":8361,"threads_started":1} W0506 07:57:56.065392 14326 catalog_manager.cc:1036] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:56.065610 14326 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:56.067538 14693 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:55.934028 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:55.943461 (+ 9433us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:55.943537 (+ 76us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:56.008222 (+ 64685us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:56.009222 (+ 1000us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:56.009259 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:56.011184 (+ 1925us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:56.011464 (+ 280us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.012326 (+ 862us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.012379 (+ 53us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.022626 (+ 10247us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.022685 (+ 59us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.024514 (+ 1829us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.024599 (+ 85us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.026937 (+ 2338us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.026967 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.027397 (+ 430us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.028466 (+ 1069us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:56.028487 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:56.032293 (+ 3806us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:56.041745 (+ 9452us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.041774 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.041800 (+ 26us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.052835 (+ 11035us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.056997 (+ 4162us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.057021 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.057039 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.057291 (+ 252us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.060027 (+ 2736us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:56.060054 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:56.060780 (+ 726us) client_negotiation.cc:695] Sending connection context 0506 07:57:56.061349 (+ 569us) client_negotiation.cc:225] Negotiation successful 0506 07:57:56.061760 (+ 411us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9026,"mutex_wait_us":10297,"thread_start_us":17920,"threads_started":1} I0506 07:57:56.073611 14700 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:56.000835 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:56.023224 (+ 22389us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:56.023312 (+ 88us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:56.023442 (+ 130us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:56.024400 (+ 958us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:56.024425 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:56.025194 (+ 769us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:56.025436 (+ 242us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.025908 (+ 472us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.025971 (+ 63us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.027029 (+ 1058us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.027055 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.029094 (+ 2039us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.029161 (+ 67us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.031531 (+ 2370us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.031568 (+ 37us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.031992 (+ 424us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.032837 (+ 845us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:56.032856 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:56.043810 (+ 10954us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:56.053717 (+ 9907us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.053744 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.053764 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.054923 (+ 1159us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.059195 (+ 4272us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.059223 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.059242 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.059722 (+ 480us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.070325 (+ 10603us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:56.070352 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:56.070689 (+ 337us) client_negotiation.cc:695] Sending connection context 0506 07:57:56.071111 (+ 422us) client_negotiation.cc:225] Negotiation successful 0506 07:57:56.071429 (+ 318us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":21936,"mutex_wait_us":821,"thread_start_us":12752,"threads_started":1} I0506 07:57:56.082031 14696 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:55.969814 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:55.992258 (+ 22444us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:55.992460 (+ 202us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:56.009194 (+ 16734us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:56.009827 (+ 633us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:56.009854 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:56.014187 (+ 4333us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:56.023287 (+ 9100us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.023729 (+ 442us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.023776 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.025786 (+ 2010us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.025814 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.034072 (+ 8258us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:56.034118 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.040577 (+ 6459us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.040609 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:56.041112 (+ 503us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.041528 (+ 416us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:56.041550 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:56.048300 (+ 6750us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:56.058248 (+ 9948us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.058275 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.058296 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.059941 (+ 1645us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.077061 (+ 17120us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:56.077089 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:56.077110 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:56.077276 (+ 166us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:56.079188 (+ 1912us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:56.079212 (+ 24us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:56.079490 (+ 278us) client_negotiation.cc:695] Sending connection context 0506 07:57:56.079887 (+ 397us) client_negotiation.cc:225] Negotiation successful 0506 07:57:56.080169 (+ 282us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":21967,"mutex_wait_us":4387,"thread_start_us":21392,"threads_started":1} I0506 07:57:56.082464 14699 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:56.001058 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:33902 0506 07:57:56.007448 (+ 6390us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:56.007473 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:56.023546 (+ 16073us) server_negotiation.cc:373] Connection header received 0506 07:57:56.023808 (+ 262us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:56.023832 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:56.023958 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:56.024094 (+ 136us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:56.026188 (+ 2094us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.026742 (+ 554us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.029391 (+ 2649us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.030873 (+ 1482us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.031092 (+ 219us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.045963 (+ 14871us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:56.046038 (+ 75us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:56.046064 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:56.046145 (+ 81us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:56.052919 (+ 6774us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.057602 (+ 4683us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.057659 (+ 57us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.057678 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.058494 (+ 816us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.068705 (+ 10211us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.068732 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.068751 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.069672 (+ 921us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:56.075142 (+ 5470us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:56.075562 (+ 420us) server_negotiation.cc:264] Negotiation successful 0506 07:57:56.075938 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6180,"thread_start_us":5946,"threads_started":1} I0506 07:57:56.082876 14695 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:55.965926 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.66:45448 0506 07:57:55.980802 (+ 14876us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:55.980834 (+ 32us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:55.992728 (+ 11894us) server_negotiation.cc:373] Connection header received 0506 07:57:56.009399 (+ 16671us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:56.009427 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:56.009555 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:56.009760 (+ 205us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:56.024962 (+ 15202us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.025491 (+ 529us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.034332 (+ 8841us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:56.035921 (+ 1589us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:56.036103 (+ 182us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:56.049111 (+ 13008us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:56.049185 (+ 74us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:56.049212 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:56.049291 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:56.056526 (+ 7235us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.075844 (+ 19318us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.075870 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.075888 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.076519 (+ 631us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:56.078185 (+ 1666us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:56.078211 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:56.078230 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:56.078725 (+ 495us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:56.079109 (+ 384us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:56.080205 (+ 1096us) server_negotiation.cc:264] Negotiation successful 0506 07:57:56.080524 (+ 319us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14567,"thread_start_us":16189,"threads_started":1} I0506 07:57:56.085630 14694 heartbeater.cc:346] Connected to a master server at 127.0.196.124:43659 I0506 07:57:56.086068 14694 heartbeater.cc:426] Registering TS with master... I0506 07:57:56.087170 14694 heartbeater.cc:475] Master 127.0.196.124:43659 requested a full tablet report, sending... I0506 07:57:56.089936 14201 ts_manager.cc:81] Registered new tserver with Master: 9e5d664019fd4fecb8189e7e5afc0f8a (127.0.196.66:41644) W0506 07:57:56.149684 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:56.149904 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:56.074409 14689 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50795 I0506 07:57:56.150678 14689 heartbeater.cc:426] Registering TS with master... I0506 07:57:56.151178 14689 heartbeater.cc:475] Master 127.0.196.126:50795 requested a full tablet report, sending... I0506 07:57:56.153959 13819 ts_manager.cc:81] Registered new tserver with Master: 9e5d664019fd4fecb8189e7e5afc0f8a (127.0.196.66:41644) I0506 07:57:56.086134 14691 heartbeater.cc:346] Connected to a master server at 127.0.196.125:45294 I0506 07:57:56.156350 14691 heartbeater.cc:426] Registering TS with master... I0506 07:57:56.156903 14691 heartbeater.cc:475] Master 127.0.196.125:45294 requested a full tablet report, sending... I0506 07:57:56.159921 14039 ts_manager.cc:81] Registered new tserver with Master: 9e5d664019fd4fecb8189e7e5afc0f8a (127.0.196.66:41644) WARNING: Logging before InitGoogleLogging() is written to STDERR W0506 07:57:56.291230 14701 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0506 07:57:56.291720 14701 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.0.196.67 W0506 07:57:56.292237 14701 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0506 07:57:56.309371 14701 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0506 07:57:56.309588 14701 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0506 07:57:56.309890 14701 flags.cc:412] Enabled experimental flag: --principal=kudu/127.0.196.67 W0506 07:57:56.310425 14701 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.196.67 I0506 07:57:56.369771 14701 tablet_server_main.cc:72] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.196.67:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/krb5kdc/kudu_127.0.196.67.keytab --principal=kudu/127.0.196.67 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.196.67 --webserver_port=0 --tserver_master_addrs=127.0.196.126:50795,127.0.196.125:45294,127.0.196.124:43659 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.196.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 8e11defa118d88551ad4b8837f73c9ca01f6907f build type FASTDEBUG built by jenkins at 06 May 2018 07:24:22 PST on quasar-rkgotj-1.vpc.cloudera.com build id 50207 TSAN enabled I0506 07:57:56.403300 14701 tablet_server_main.cc:79] Initializing tablet server... W0506 07:57:56.404448 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:56.404666 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:56.413453 14701 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 233678us May 06 07:57:56 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0506 07:57:56.440975 14701 init.cc:339] Logged in from keytab as kudu/127.0.196.67@KRBTEST.COM (short username kudu) I0506 07:57:56.456953 14701 fs_manager.cc:260] Metadata directory not provided I0506 07:57:56.457150 14701 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0506 07:57:56.457466 14701 server_base.cc:433] Could not load existing FS layout: Not found: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0506 07:57:56.457653 14701 server_base.cc:434] Attempting to create new FS layout instead I0506 07:57:56.481277 14701 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/instance: uuid: "4b8171dd3cac4a02892d3fa117f2b69d" format_stamp: "Formatted at 2018-05-06 14:57:56 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:56.482498 14701 fs_manager.cc:595] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/wal/instance: uuid: "4b8171dd3cac4a02892d3fa117f2b69d" format_stamp: "Formatted at 2018-05-06 14:57:56 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:56.513177 14701 fs_manager.cc:495] Time spent creating directory manager: real 0.030s user 0.013s sys 0.004s I0506 07:57:56.515177 14701 env_posix.cc:1643] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0506 07:57:56.516785 14701 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0506 07:57:56.551180 14701 fs_manager.cc:417] Time spent opening block manager: real 0.015s user 0.001s sys 0.001s I0506 07:57:56.551364 14701 fs_manager.cc:428] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/wal uuid: "4b8171dd3cac4a02892d3fa117f2b69d" format_stamp: "Formatted at 2018-05-06 14:57:56 on quasar-rkgotj-1.vpc.cloudera.com" I0506 07:57:56.551710 14701 fs_report.cc:347] Block manager report -------------------- 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0506 07:57:57.054922 14701 env_posix.cc:1638] Raising this process' running threads per effective uid limit from 32768 to 65536 I0506 07:57:57.063189 14701 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) W0506 07:57:57.067940 14326 catalog_manager.cc:1036] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:57.068151 14326 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:57.069813 14701 tablet_server_main.cc:84] Starting tablet server... W0506 07:57:57.152256 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:57.152505 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0506 07:57:57.407048 14358 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:57.407255 14358 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:57.724524 14701 rpc_server.cc:205] RPC server started. Bound to: 127.0.196.67:55227 I0506 07:57:57.726300 14701 webserver.cc:173] Starting webserver on 127.0.196.67:0 I0506 07:57:57.726565 14701 webserver.cc:184] Document root disabled I0506 07:57:57.732515 14701 webserver.cc:311] Webserver started. Bound to: http://127.0.196.67:33630/ I0506 07:57:57.733589 14701 server_base.cc:610] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1525618592290500-785/minicluster-data/ts-2/data/info.pb I0506 07:57:57.737433 785 external_mini_cluster.cc:848] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 14701 I0506 07:57:57.769276 14701 tablet_server_main.cc:87] Tablet server successfully started. May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM I0506 07:57:57.803670 14918 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.738771 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38430 0506 07:57:57.758517 (+ 19746us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.758542 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.758582 (+ 40us) server_negotiation.cc:373] Connection header received 0506 07:57:57.758713 (+ 131us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.758737 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.758854 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.758993 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.761929 (+ 2936us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.762469 (+ 540us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.764547 (+ 2078us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.766072 (+ 1525us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.766250 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.774366 (+ 8116us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.774398 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.774425 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.774499 (+ 74us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.791685 (+ 17186us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.795304 (+ 3619us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.795331 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.795350 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.796027 (+ 677us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.797688 (+ 1661us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.797716 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.797735 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.798231 (+ 496us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.801190 (+ 2959us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.801393 (+ 203us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.801867 (+ 474us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19546,"thread_start_us":19410,"threads_started":1} May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.124@KRBTEST.COM May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.196.67@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:57.846525 14924 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.751848 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:42713 0506 07:57:57.761461 (+ 9613us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.761485 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.770324 (+ 8839us) server_negotiation.cc:373] Connection header received 0506 07:57:57.809241 (+ 38917us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.809270 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.809401 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.809555 (+ 154us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.811015 (+ 1460us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.811570 (+ 555us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.813931 (+ 2361us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.815421 (+ 1490us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.815605 (+ 184us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.821106 (+ 5501us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.821178 (+ 72us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.821206 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.821288 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.828353 (+ 7065us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.834128 (+ 5775us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.834155 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.834173 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.835372 (+ 1199us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.837842 (+ 2470us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.837869 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.837888 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.838382 (+ 494us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.838801 (+ 419us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.844406 (+ 5605us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.844756 (+ 350us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":9415,"thread_start_us":9773,"threads_started":1} I0506 07:57:57.848454 14925 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.752018 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.125:45294 0506 07:57:57.770023 (+ 18005us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:57.770163 (+ 140us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:57.809031 (+ 38868us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:57.809620 (+ 589us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:57.809676 (+ 56us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:57.810280 (+ 604us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:57.810505 (+ 225us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.810805 (+ 300us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.810850 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.811771 (+ 921us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.811798 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.813713 (+ 1915us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.813757 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.816490 (+ 2733us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.816523 (+ 33us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.816997 (+ 474us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.817278 (+ 281us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:57.817304 (+ 26us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:57.820280 (+ 2976us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:57.830290 (+ 10010us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.830321 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.830346 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.833606 (+ 3260us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.836363 (+ 2757us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.836389 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.836408 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.836692 (+ 284us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.843077 (+ 6385us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:57.843107 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:57.843509 (+ 402us) client_negotiation.cc:695] Sending connection context 0506 07:57:57.844050 (+ 541us) client_negotiation.cc:225] Negotiation successful 0506 07:57:57.844426 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":17765,"mutex_wait_us":6013,"thread_start_us":14927,"threads_started":1} May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.125@KRBTEST.COM I0506 07:57:57.851755 14920 heartbeater.cc:346] Connected to a master server at 127.0.196.125:45294 I0506 07:57:57.852052 14920 heartbeater.cc:426] Registering TS with master... I0506 07:57:57.852639 14920 heartbeater.cc:475] Master 127.0.196.125:45294 requested a full tablet report, sending... I0506 07:57:57.855554 14039 ts_manager.cc:81] Registered new tserver with Master: 4b8171dd3cac4a02892d3fa117f2b69d (127.0.196.67:55227) I0506 07:57:57.859542 14928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.779987 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:53817 0506 07:57:57.788303 (+ 8316us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.788332 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.795431 (+ 7099us) server_negotiation.cc:373] Connection header received 0506 07:57:57.806499 (+ 11068us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.806529 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.806664 (+ 135us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.806808 (+ 144us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.813984 (+ 7176us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.814537 (+ 553us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.818940 (+ 4403us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.820483 (+ 1543us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.820675 (+ 192us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.829154 (+ 8479us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.829235 (+ 81us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.829260 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.829335 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.836222 (+ 6887us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.841502 (+ 5280us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.841533 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.841551 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.842201 (+ 650us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.845222 (+ 3021us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.845248 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.845267 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.845782 (+ 515us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.846159 (+ 377us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.857240 (+ 11081us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.857664 (+ 424us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8094,"thread_start_us":7811,"threads_started":1} I0506 07:57:57.865679 14927 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.778903 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.124:43659 0506 07:57:57.794766 (+ 15863us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:57.794908 (+ 142us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:57.806291 (+ 11383us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:57.807285 (+ 994us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:57.807318 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:57.808173 (+ 855us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:57.808477 (+ 304us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.808834 (+ 357us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.808883 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.814843 (+ 5960us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.814871 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.816172 (+ 1301us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.816216 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.821116 (+ 4900us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.821143 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.821537 (+ 394us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.821771 (+ 234us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:57.821789 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:57.824867 (+ 3078us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:57.837912 (+ 13045us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.837938 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.837959 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.840849 (+ 2890us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.844314 (+ 3465us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.844341 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.844362 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.844532 (+ 170us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.853244 (+ 8712us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:57.853272 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:57.853603 (+ 331us) client_negotiation.cc:695] Sending connection context 0506 07:57:57.854284 (+ 681us) client_negotiation.cc:225] Negotiation successful 0506 07:57:57.854600 (+ 316us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15636,"mutex_wait_us":5947,"thread_start_us":15982,"threads_started":1} I0506 07:57:57.869292 14923 heartbeater.cc:346] Connected to a master server at 127.0.196.124:43659 I0506 07:57:57.869524 14923 heartbeater.cc:426] Registering TS with master... I0506 07:57:57.869990 14923 heartbeater.cc:475] Master 127.0.196.124:43659 requested a full tablet report, sending... I0506 07:57:57.872467 14201 ts_manager.cc:81] Registered new tserver with Master: 4b8171dd3cac4a02892d3fa117f2b69d (127.0.196.67:55227) I0506 07:57:57.872563 14922 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.745274 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.67:57342 0506 07:57:57.765394 (+ 20120us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.765427 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.765478 (+ 51us) server_negotiation.cc:373] Connection header received 0506 07:57:57.806262 (+ 40784us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.806289 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.806409 (+ 120us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.806540 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.812280 (+ 5740us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.812878 (+ 598us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.833605 (+ 20727us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.835217 (+ 1612us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.835400 (+ 183us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.840558 (+ 5158us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.840635 (+ 77us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.840662 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.840775 (+ 113us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.847919 (+ 7144us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.857767 (+ 9848us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.857795 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.857815 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.858475 (+ 660us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.867062 (+ 8587us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.867089 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.867108 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.867598 (+ 490us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.868021 (+ 423us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.870364 (+ 2343us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.870712 (+ 348us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19805,"thread_start_us":6354,"threads_started":1} I0506 07:57:57.875566 14921 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.743728 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.0.196.126:50795 0506 07:57:57.745761 (+ 2033us) negotiation.cc:98] Waiting for socket to connect 0506 07:57:57.745840 (+ 79us) client_negotiation.cc:160] Beginning negotiation 0506 07:57:57.805975 (+ 60135us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0506 07:57:57.809267 (+ 3292us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0506 07:57:57.809294 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0506 07:57:57.811387 (+ 2093us) client_negotiation.cc:175] Negotiated authn=SASL 0506 07:57:57.811565 (+ 178us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.812069 (+ 504us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.812112 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.822503 (+ 10391us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.822532 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.833333 (+ 10801us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0506 07:57:57.833384 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.836327 (+ 2943us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.836361 (+ 34us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0506 07:57:57.836880 (+ 519us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.837125 (+ 245us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0506 07:57:57.837143 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0506 07:57:57.839756 (+ 2613us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0506 07:57:57.849755 (+ 9999us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.849782 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.849804 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.851043 (+ 1239us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.865936 (+ 14893us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0506 07:57:57.865963 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0506 07:57:57.865983 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0506 07:57:57.866145 (+ 162us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0506 07:57:57.868813 (+ 2668us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0506 07:57:57.868839 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0506 07:57:57.869151 (+ 312us) client_negotiation.cc:695] Sending connection context 0506 07:57:57.869621 (+ 470us) client_negotiation.cc:225] Negotiation successful 0506 07:57:57.873397 (+ 3776us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1660,"mutex_wait_us":7572,"thread_start_us":2037,"threads_started":1} I0506 07:57:57.876740 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.817982 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47374 0506 07:57:57.825467 (+ 7485us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.825494 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.825532 (+ 38us) server_negotiation.cc:373] Connection header received 0506 07:57:57.825661 (+ 129us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.825686 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.825797 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.828185 (+ 2388us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.836872 (+ 8687us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.837518 (+ 646us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.842807 (+ 5289us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.846667 (+ 3860us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.851173 (+ 4506us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.851491 (+ 318us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.851519 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.851543 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.851615 (+ 72us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.865426 (+ 13811us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.868505 (+ 3079us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.868539 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.868558 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.869848 (+ 1290us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.871127 (+ 1279us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.871154 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.871174 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.871695 (+ 521us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.874268 (+ 2573us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.874488 (+ 220us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.874966 (+ 478us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2262,"server-negotiator.queue_time_us":7158,"thread_start_us":6001,"threads_started":1} I0506 07:57:57.878315 14917 heartbeater.cc:346] Connected to a master server at 127.0.196.126:50795 I0506 07:57:57.878549 14917 heartbeater.cc:426] Registering TS with master... I0506 07:57:57.878999 14917 heartbeater.cc:475] Master 127.0.196.126:50795 requested a full tablet report, sending... I0506 07:57:57.882388 13819 ts_manager.cc:81] Registered new tserver with Master: 4b8171dd3cac4a02892d3fa117f2b69d (127.0.196.67:55227) May 06 07:57:57 quasar-rkgotj-1.vpc.cloudera.com krb5kdc[13533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1525618668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.196.126@KRBTEST.COM I0506 07:57:57.926249 14922 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.880724 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:53028 0506 07:57:57.881016 (+ 292us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.881040 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.881077 (+ 37us) server_negotiation.cc:373] Connection header received 0506 07:57:57.881176 (+ 99us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.881201 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.881314 (+ 113us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.881437 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:57.883584 (+ 2147us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.884250 (+ 666us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.886104 (+ 1854us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:57.887679 (+ 1575us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:57.887860 (+ 181us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:57.903256 (+ 15396us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:57.903284 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:57.903307 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:57.903379 (+ 72us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:57.909984 (+ 6605us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.920017 (+ 10033us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.920045 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.920062 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.920690 (+ 628us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:57.921302 (+ 612us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:57.921327 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:57.921345 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:57.921853 (+ 508us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:57.923756 (+ 1903us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:57.923957 (+ 201us) server_negotiation.cc:264] Negotiation successful 0506 07:57:57.924409 (+ 452us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":101} I0506 07:57:57.931798 785 external_mini_cluster.cc:482] 3 TS(s) registered with all masters I0506 07:57:57.933181 13844 tablet_service.cc:1113] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000" dest_uuid: "d4d2993be16d445c8c7f502179e16cb5" from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.196.126:53028 I0506 07:57:57.933682 13844 raft_consensus.cc:435] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0506 07:57:57.933949 13844 raft_consensus.cc:2756] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:57.938357 13844 raft_consensus.cc:457] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:57.939697 13844 leader_election.cc:230] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [CANDIDATE]: Term 1 election: Requesting vote from peer 8671bbcf68254398acccde133788f79a I0506 07:57:57.940240 13844 leader_election.cc:230] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [CANDIDATE]: Term 1 election: Requesting vote from peer 731d40aa19f0477cbf46586776702a61 I0506 07:57:57.941857 14050 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4d2993be16d445c8c7f502179e16cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8671bbcf68254398acccde133788f79a" I0506 07:57:57.942438 14050 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:57.945009 14223 tablet_service.cc:998] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4d2993be16d445c8c7f502179e16cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "731d40aa19f0477cbf46586776702a61" I0506 07:57:57.945578 14223 raft_consensus.cc:2756] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [term 0 FOLLOWER]: Advancing to term 1 I0506 07:57:57.946903 14050 raft_consensus.cc:2219] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d2993be16d445c8c7f502179e16cb5 in term 1. I0506 07:57:57.947793 13679 leader_election.cc:362] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [CANDIDATE]: Term 1 election: Vote granted by peer 8671bbcf68254398acccde133788f79a I0506 07:57:57.948020 13679 leader_election.cc:257] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0506 07:57:57.949838 14932 raft_consensus.cc:2511] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 1 FOLLOWER]: Leader election won for term 1 I0506 07:57:57.953125 14223 raft_consensus.cc:2219] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4d2993be16d445c8c7f502179e16cb5 in term 1. I0506 07:57:57.953958 13679 leader_election.cc:362] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [CANDIDATE]: Term 1 election: Vote granted by peer 731d40aa19f0477cbf46586776702a61 I0506 07:57:57.955761 14932 raft_consensus.cc:571] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [term 1 LEADER]: Becoming Leader. State: Replica: d4d2993be16d445c8c7f502179e16cb5, State: Running, Role: LEADER I0506 07:57:57.956226 14932 consensus_queue.cc:204] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } I0506 07:57:57.962996 14933 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d2993be16d445c8c7f502179e16cb5. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:57.963695 14933 sys_catalog.cc:338] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: This master's current role is: LEADER I0506 07:57:57.972936 14937 catalog_manager.cc:990] Loading table and tablet metadata into memory... I0506 07:57:57.977046 14937 catalog_manager.cc:1001] Initializing Kudu internal certificate authority... I0506 07:57:58.032238 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.981450 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47377 0506 07:57:57.982028 (+ 578us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.982050 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.987361 (+ 5311us) server_negotiation.cc:373] Connection header received 0506 07:57:57.987796 (+ 435us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.987821 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.987948 (+ 127us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.988078 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.000507 (+ 12429us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.001049 (+ 542us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.007934 (+ 6885us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.009398 (+ 1464us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.009584 (+ 186us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.013812 (+ 4228us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:58.013842 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:58.013865 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:58.013944 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:58.020420 (+ 6476us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.025772 (+ 5352us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.025799 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.025820 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.026549 (+ 729us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.028055 (+ 1506us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.028084 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.028103 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.028605 (+ 502us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:58.028916 (+ 311us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.029848 (+ 932us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.030371 (+ 523us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":390} I0506 07:57:58.060380 14928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.987920 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38435 0506 07:57:57.988463 (+ 543us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.988487 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.001116 (+ 12629us) server_negotiation.cc:373] Connection header received 0506 07:57:58.002769 (+ 1653us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.002796 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.002915 (+ 119us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.003039 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.008443 (+ 5404us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.009033 (+ 590us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.015901 (+ 6868us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.017394 (+ 1493us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.017569 (+ 175us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.028418 (+ 10849us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:58.028446 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:58.028470 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:58.028547 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:58.035498 (+ 6951us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.045355 (+ 9857us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.045381 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.045399 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.046051 (+ 652us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.049118 (+ 3067us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.049145 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.049164 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.049790 (+ 626us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:58.050108 (+ 318us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.052686 (+ 2578us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.053235 (+ 549us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":319} I0506 07:57:58.061331 14922 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:57.981447 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:53029 0506 07:57:57.981792 (+ 345us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:57.981814 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:57.987649 (+ 5835us) server_negotiation.cc:373] Connection header received 0506 07:57:57.988560 (+ 911us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:57.988613 (+ 53us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:57.988745 (+ 132us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:57.988878 (+ 133us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.008404 (+ 19526us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.008944 (+ 540us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.017972 (+ 9028us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.019465 (+ 1493us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.019649 (+ 184us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.036840 (+ 17191us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0506 07:57:58.036872 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0506 07:57:58.036901 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0506 07:57:58.036989 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0506 07:57:58.043493 (+ 6504us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.047755 (+ 4262us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.047783 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.047802 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.048461 (+ 659us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0506 07:57:58.050666 (+ 2205us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0506 07:57:58.050722 (+ 56us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0506 07:57:58.050744 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0506 07:57:58.051262 (+ 518us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0506 07:57:58.051553 (+ 291us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.054824 (+ 3271us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.055292 (+ 468us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":126} I0506 07:57:58.152037 14223 raft_consensus.cc:1057] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [term 1 FOLLOWER]: Refusing update from remote peer d4d2993be16d445c8c7f502179e16cb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:58.152978 14050 raft_consensus.cc:1057] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [term 1 FOLLOWER]: Refusing update from remote peer d4d2993be16d445c8c7f502179e16cb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0506 07:57:58.153412 14932 consensus_queue.cc:963] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0506 07:57:58.154803 14328 catalog_manager.cc:1036] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0506 07:57:58.155001 14328 catalog_manager.cc:562] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0506 07:57:58.155431 14932 consensus_queue.cc:963] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0506 07:57:58.183810 14975 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d2993be16d445c8c7f502179e16cb5. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.184492 14975 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:58.185207 14974 sys_catalog.cc:335] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d2993be16d445c8c7f502179e16cb5. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.185932 14974 sys_catalog.cc:338] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:58.191923 14974 sys_catalog.cc:335] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.192575 14974 sys_catalog.cc:338] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61 [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:58.193616 14932 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.194423 14932 sys_catalog.cc:338] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: This master's current role is: LEADER I0506 07:57:58.195791 14975 sys_catalog.cc:335] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.196426 14975 sys_catalog.cc:338] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a [sys.catalog]: This master's current role is: FOLLOWER I0506 07:57:58.200713 14937 catalog_manager.cc:888] Generated new certificate authority record I0506 07:57:58.203261 14937 catalog_manager.cc:1010] Loading token signing keys... I0506 07:57:58.214730 14932 sys_catalog.cc:335] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4d2993be16d445c8c7f502179e16cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4d2993be16d445c8c7f502179e16cb5" member_type: VOTER last_known_addr { host: "127.0.196.126" port: 50795 } } peers { permanent_uuid: "8671bbcf68254398acccde133788f79a" member_type: VOTER last_known_addr { host: "127.0.196.125" port: 45294 } } peers { permanent_uuid: "731d40aa19f0477cbf46586776702a61" member_type: VOTER last_known_addr { host: "127.0.196.124" port: 43659 } } } I0506 07:57:58.215368 14932 sys_catalog.cc:338] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5 [sys.catalog]: This master's current role is: LEADER I0506 07:57:58.248747 14937 catalog_manager.cc:3992] T 00000000000000000000000000000000 P d4d2993be16d445c8c7f502179e16cb5: Generated new TSK 0 W0506 07:57:58.385957 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.371142 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38438 0506 07:57:58.371478 (+ 336us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.371498 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.382808 (+ 11310us) server_negotiation.cc:373] Connection header received 0506 07:57:58.383026 (+ 218us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.383051 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.383171 (+ 120us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.383295 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.385084 (+ 1789us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38438: BlockingRecv error: recv got EOF from 127.0.196.124:38438 (error 108) Metrics: {"server-negotiator.queue_time_us":134} W0506 07:57:58.411166 14922 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.369745 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:53032 0506 07:57:58.370133 (+ 388us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.370157 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.373954 (+ 3797us) server_negotiation.cc:373] Connection header received 0506 07:57:58.374412 (+ 458us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.374436 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.374565 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.374729 (+ 164us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.403841 (+ 29112us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.126:53032: BlockingRecv error: recv got EOF from 127.0.196.126:53032 (error 108) Metrics: {"server-negotiator.queue_time_us":115} I0506 07:57:58.416501 14358 catalog_manager.cc:1034] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: acquiring CA information for follower catalog manager: success I0506 07:57:58.419257 14358 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 8671bbcf68254398acccde133788f79a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 W0506 07:57:58.420387 14929 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.367784 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47380 0506 07:57:58.368063 (+ 279us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.368085 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.378018 (+ 9933us) server_negotiation.cc:373] Connection header received 0506 07:57:58.378355 (+ 337us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.378381 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.378499 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.378614 (+ 115us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.389439 (+ 10825us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.125:47380: BlockingRecv error: recv got EOF from 127.0.196.125:47380 (error 108) Metrics: {"server-negotiator.queue_time_us":75} I0506 07:57:58.455284 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.423149 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47382 0506 07:57:58.423442 (+ 293us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.423462 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.432039 (+ 8577us) server_negotiation.cc:373] Connection header received 0506 07:57:58.441117 (+ 9078us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.441145 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.441329 (+ 184us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.441469 (+ 140us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.446562 (+ 5093us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.447176 (+ 614us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.449593 (+ 2417us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.451091 (+ 1498us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.451264 (+ 173us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.452985 (+ 1721us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.453405 (+ 420us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.453521 (+ 116us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.453922 (+ 401us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.454113 (+ 191us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":110} W0506 07:57:58.521486 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.505737 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38440 0506 07:57:58.506039 (+ 302us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.506064 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.513715 (+ 7651us) server_negotiation.cc:373] Connection header received 0506 07:57:58.514019 (+ 304us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.514037 (+ 18us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.514173 (+ 136us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.514297 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.520773 (+ 6476us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38440: BlockingRecv error: recv got EOF from 127.0.196.124:38440 (error 108) Metrics: {"server-negotiator.queue_time_us":104} I0506 07:57:58.557240 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.533660 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47384 0506 07:57:58.533976 (+ 316us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.533997 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.541213 (+ 7216us) server_negotiation.cc:373] Connection header received 0506 07:57:58.541658 (+ 445us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.541685 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.541843 (+ 158us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.541960 (+ 117us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.543612 (+ 1652us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.544232 (+ 620us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.546487 (+ 2255us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.548021 (+ 1534us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.548199 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.554872 (+ 6673us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.555113 (+ 241us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.555212 (+ 99us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.555732 (+ 520us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.555930 (+ 198us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":100} W0506 07:57:58.594175 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.584021 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38442 0506 07:57:58.584323 (+ 302us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.584345 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.590978 (+ 6633us) server_negotiation.cc:373] Connection header received 0506 07:57:58.591263 (+ 285us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.591288 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.591433 (+ 145us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.591568 (+ 135us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.593416 (+ 1848us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38442: BlockingRecv error: recv got EOF from 127.0.196.124:38442 (error 108) Metrics: {"server-negotiator.queue_time_us":103} I0506 07:57:58.645191 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.618677 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47386 0506 07:57:58.619654 (+ 977us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.619678 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.632046 (+ 12368us) server_negotiation.cc:373] Connection header received 0506 07:57:58.632320 (+ 274us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.632345 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.632489 (+ 144us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.632618 (+ 129us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.633439 (+ 821us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.634389 (+ 950us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.640463 (+ 6074us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.641837 (+ 1374us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.642004 (+ 167us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.643135 (+ 1131us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.643354 (+ 219us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.643444 (+ 90us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.643864 (+ 420us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.644048 (+ 184us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":741} W0506 07:57:58.703315 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.692825 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38444 0506 07:57:58.693283 (+ 458us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.693310 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.700395 (+ 7085us) server_negotiation.cc:373] Connection header received 0506 07:57:58.700520 (+ 125us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.700542 (+ 22us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.700706 (+ 164us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.700834 (+ 128us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.702564 (+ 1730us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38444: BlockingRecv error: recv got EOF from 127.0.196.124:38444 (error 108) Metrics: {"server-negotiator.queue_time_us":237} I0506 07:57:58.729995 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.715868 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47388 0506 07:57:58.716132 (+ 264us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.716153 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.721128 (+ 4975us) server_negotiation.cc:373] Connection header received 0506 07:57:58.721403 (+ 275us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.721428 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.721572 (+ 144us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.721718 (+ 146us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.722540 (+ 822us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.723185 (+ 645us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.725378 (+ 2193us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.726895 (+ 1517us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.727073 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.728077 (+ 1004us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.728289 (+ 212us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.728390 (+ 101us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.728683 (+ 293us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.728871 (+ 188us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":72} W0506 07:57:58.777263 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.750497 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38446 0506 07:57:58.750816 (+ 319us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.750839 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.773307 (+ 22468us) server_negotiation.cc:373] Connection header received 0506 07:57:58.773561 (+ 254us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.773587 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.773765 (+ 178us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.773897 (+ 132us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.776513 (+ 2616us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38446: BlockingRecv error: recv got EOF from 127.0.196.124:38446 (error 108) Metrics: {"server-negotiator.queue_time_us":90} I0506 07:57:58.834254 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.813447 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47390 0506 07:57:58.813820 (+ 373us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.813842 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.822601 (+ 8759us) server_negotiation.cc:373] Connection header received 0506 07:57:58.823155 (+ 554us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.823179 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.823343 (+ 164us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.823451 (+ 108us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.824755 (+ 1304us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.825361 (+ 606us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.827879 (+ 2518us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.829371 (+ 1492us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.829533 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.832263 (+ 2730us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.832469 (+ 206us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.832558 (+ 89us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.832960 (+ 402us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.833161 (+ 201us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":181} W0506 07:57:58.883929 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:58.874026 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38448 0506 07:57:58.874328 (+ 302us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.874351 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.878671 (+ 4320us) server_negotiation.cc:373] Connection header received 0506 07:57:58.878952 (+ 281us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.878974 (+ 22us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.879102 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.879211 (+ 109us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:58.883215 (+ 4004us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38448: BlockingRecv error: recv got EOF from 127.0.196.124:38448 (error 108) Metrics: {"server-negotiator.queue_time_us":115} I0506 07:57:58.889329 13819 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.67@KRBTEST.COM'} at 127.0.196.67:57342 I0506 07:57:58.908859 13819 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.65@KRBTEST.COM'} at 127.0.196.65:34568 I0506 07:57:58.957139 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:58.933832 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47392 0506 07:57:58.934128 (+ 296us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:58.934150 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:58.940476 (+ 6326us) server_negotiation.cc:373] Connection header received 0506 07:57:58.940930 (+ 454us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:58.940956 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:58.941109 (+ 153us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:58.941228 (+ 119us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:58.942293 (+ 1065us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.942913 (+ 620us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.952116 (+ 9203us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:58.953587 (+ 1471us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:58.953798 (+ 211us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:58.955007 (+ 1209us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:58.955231 (+ 224us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:58.955329 (+ 98us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:58.955762 (+ 433us) server_negotiation.cc:264] Negotiation successful 0506 07:57:58.955951 (+ 189us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":103} W0506 07:57:59.020423 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:59.015949 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38450 0506 07:57:59.016268 (+ 319us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.016288 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.017529 (+ 1241us) server_negotiation.cc:373] Connection header received 0506 07:57:59.017821 (+ 292us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.017837 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.017934 (+ 97us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.018042 (+ 108us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:59.019683 (+ 1641us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38450: BlockingRecv error: recv got EOF from 127.0.196.124:38450 (error 108) Metrics: {"server-negotiator.queue_time_us":126} I0506 07:57:59.108345 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.087360 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47395 0506 07:57:59.087700 (+ 340us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.087721 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.096663 (+ 8942us) server_negotiation.cc:373] Connection header received 0506 07:57:59.096965 (+ 302us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.096989 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.097133 (+ 144us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.097248 (+ 115us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.098128 (+ 880us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.098715 (+ 587us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.103960 (+ 5245us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.105355 (+ 1395us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.105524 (+ 169us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.106422 (+ 898us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.106647 (+ 225us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.106749 (+ 102us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.106995 (+ 246us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.107178 (+ 183us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":125} I0506 07:57:59.161566 14328 catalog_manager.cc:1034] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: acquiring CA information for follower catalog manager: success W0506 07:57:59.164266 14928 negotiation.cc:313] Failed RPC negotiation. Trace: 0506 07:57:59.152986 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38453 0506 07:57:59.153247 (+ 261us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.153267 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.159638 (+ 6371us) server_negotiation.cc:373] Connection header received 0506 07:57:59.160015 (+ 377us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.160039 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.160189 (+ 150us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.160315 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0506 07:57:59.163513 (+ 3198us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.196.124:38453: BlockingRecv error: recv got EOF from 127.0.196.124:38453 (error 108) Metrics: {"server-negotiator.queue_time_us":74} I0506 07:57:59.165946 14328 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 731d40aa19f0477cbf46586776702a61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0506 07:57:59.167352 13819 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.196.66@KRBTEST.COM'} at 127.0.196.66:53568 I0506 07:57:59.277935 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.270018 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47397 0506 07:57:59.270295 (+ 277us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.270317 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.270974 (+ 657us) server_negotiation.cc:373] Connection header received 0506 07:57:59.271218 (+ 244us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.271234 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.271333 (+ 99us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.271430 (+ 97us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.272088 (+ 658us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.272452 (+ 364us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.273824 (+ 1372us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.274731 (+ 907us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.274897 (+ 166us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.275839 (+ 942us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.276068 (+ 229us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.276159 (+ 91us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.276561 (+ 402us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.276767 (+ 206us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":89} I0506 07:57:59.312714 14928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.302383 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38455 0506 07:57:59.302836 (+ 453us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.302857 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.305565 (+ 2708us) server_negotiation.cc:373] Connection header received 0506 07:57:59.305843 (+ 278us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.305859 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.305962 (+ 103us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.306069 (+ 107us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.306802 (+ 733us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.307151 (+ 349us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.309107 (+ 1956us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.309965 (+ 858us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.310123 (+ 158us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.311005 (+ 882us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.311298 (+ 293us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.311405 (+ 107us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.311643 (+ 238us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.311851 (+ 208us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":265} I0506 07:57:59.340168 15202 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.328197 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.126:53052 0506 07:57:59.330578 (+ 2381us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.330627 (+ 49us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.330662 (+ 35us) server_negotiation.cc:373] Connection header received 0506 07:57:59.330796 (+ 134us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.330817 (+ 21us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.330953 (+ 136us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.331067 (+ 114us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.332078 (+ 1011us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.332747 (+ 669us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.334935 (+ 2188us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.336306 (+ 1371us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.336465 (+ 159us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.337556 (+ 1091us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.337899 (+ 343us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.337995 (+ 96us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.338843 (+ 848us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.339014 (+ 171us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":1863,"threads_started":1} I0506 07:57:59.347754 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 14309 I0506 07:57:59.363906 14929 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.328330 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.125:47400 0506 07:57:59.328653 (+ 323us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.328675 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.340847 (+ 12172us) server_negotiation.cc:373] Connection header received 0506 07:57:59.341217 (+ 370us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.341240 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.341386 (+ 146us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.341503 (+ 117us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.342449 (+ 946us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.343021 (+ 572us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.355862 (+ 12841us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.357327 (+ 1465us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.357521 (+ 194us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.359708 (+ 2187us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.359958 (+ 250us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.360055 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.362483 (+ 2428us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.362710 (+ 227us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":102} I0506 07:57:59.370811 14928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0506 07:57:59.352780 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.196.124:38458 0506 07:57:59.353086 (+ 306us) server_negotiation.cc:176] Beginning negotiation 0506 07:57:59.353106 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0506 07:57:59.353842 (+ 736us) server_negotiation.cc:373] Connection header received 0506 07:57:59.354245 (+ 403us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0506 07:57:59.354269 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0506 07:57:59.354400 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0506 07:57:59.354506 (+ 106us) server_negotiation.cc:197] Negotiated authn=TOKEN 0506 07:57:59.357027 (+ 2521us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.358010 (+ 983us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.364803 (+ 6793us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0506 07:57:59.366211 (+ 1408us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0506 07:57:59.366362 (+ 151us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0506 07:57:59.368283 (+ 1921us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0506 07:57:59.368470 (+ 187us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0506 07:57:59.368551 (+ 81us) server_negotiation.cc:901] Waiting for connection context 0506 07:57:59.369589 (+ 1038us) server_negotiation.cc:264] Negotiation successful 0506 07:57:59.369781 (+ 192us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":123} W0506 07:57:59.380272 14124 connection.cc:512] server connection from 127.0.196.65:35126 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:35126): Connection reset by peer (error 104) W0506 07:57:59.380302 14001 connection.cc:512] server connection from 127.0.196.65:58455 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:58455): Connection reset by peer (error 104) I0506 07:57:59.381175 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 14549 W0506 07:57:59.385282 13688 connection.cc:512] server connection from 127.0.196.65:34568 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.65:34568): Connection reset by peer (error 104) W0506 07:57:59.418155 13760 connection.cc:512] server connection from 127.0.196.66:53568 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:53568): Connection reset by peer (error 104) W0506 07:57:59.418146 14001 connection.cc:512] server connection from 127.0.196.66:45448 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:45448): Connection reset by peer (error 104) W0506 07:57:59.419144 14160 connection.cc:512] server connection from 127.0.196.66:33902 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.66:33902): Connection reset by peer (error 104) I0506 07:57:59.420006 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 14701 W0506 07:57:59.456811 13683 connection.cc:512] server connection from 127.0.196.67:57342 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:57342): Connection reset by peer (error 104) W0506 07:57:59.457831 14001 connection.cc:512] server connection from 127.0.196.67:42713 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:42713): Connection reset by peer (error 104) W0506 07:57:59.458912 14122 connection.cc:512] server connection from 127.0.196.67:53817 recv error: Network error: failed to read from TLS socket (remote: 127.0.196.67:53817): Connection reset by peer (error 104) I0506 07:57:59.459926 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 13576 I0506 07:57:59.487448 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 13917 I0506 07:57:59.539631 785 external_mini_cluster.cc:978] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 14078 [ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (12079 ms) [----------] 2 tests from ConnectToFollowerMasterTest (22225 ms total) [----------] Global test environment tear-down [==========] 23 tests from 3 test cases ran. (87165 ms total) [ PASSED ] 23 tests. ================== WARNING: ThreadSanitizer: data race (pid=785) Write of size 8 at 0x7b0800000d68 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 8 at 0x7b0800000d68 by thread T5: #0 std::__1::__tree_end_node*>* std::__1::__tree_next_iter*>*, std::__1::__tree_node_base*>(std::__1::__tree_node_base*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:185:14 (libkrpc.so+0x90356) #1 std::__1::__tree_const_iterator*, long>::operator++() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:921 (libkrpc.so+0x90356) #2 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1667 (libkrpc.so+0x90356) #3 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #4 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #5 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #9 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #10 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #11 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #12 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0x9ed1d) #13 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1c8b3d) #14 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c4868) #15 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1cf7d9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1cf72a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1cf6b3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1cf4a9) #19 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #20 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) Thread T5 'client-negotiat' (tid=15200, finished) created by thread T3 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bb9ef) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c6b85) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c3e72) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c263f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c40df) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c4021) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:582:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:552:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:336:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:841:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:310:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=785) Write of size 8 at 0x7b0800000d70 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 8 at 0x7b0800000d70 by thread T5: #0 std::__1::__tree_end_node*>* std::__1::__tree_next_iter*>*, std::__1::__tree_node_base*>(std::__1::__tree_node_base*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:84:24 (libkrpc.so+0x9039c) #1 std::__1::__tree_const_iterator*, long>::operator++() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:921 (libkrpc.so+0x9039c) #2 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1667 (libkrpc.so+0x9039c) #3 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #4 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #5 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #9 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #10 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #11 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #12 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0x9ed1d) #13 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1c8b3d) #14 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c4868) #15 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1cf7d9) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1cf72a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1cf6b3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1cf4a9) #19 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #20 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) Thread T5 'client-negotiat' (tid=15200, finished) created by thread T3 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bb9ef) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c6b85) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c3e72) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c263f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c40df) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c4021) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:582:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:552:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:336:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:841:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:310:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=785) Write of size 8 at 0x7b0800000d78 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 4 at 0x7b0800000d7c by thread T5: #0 _ZNSt3__19allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEE9constructIS4_JRKS4_EEEvPT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1805:35 (libkrpc.so+0x90d18) #1 _ZNSt3__116allocator_traitsINS_9allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEEEE11__constructIS5_JRKS5_EEEvNS_17integral_constantIbLb1EEERS8_PT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1715 (libkrpc.so+0x90d18) #2 _ZNSt3__116allocator_traitsINS_9allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEEEE9constructIS5_JRKS5_EEEvRS8_PT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1561 (libkrpc.so+0x90d18) #3 _ZNSt3__16__treeIN4kudu3rpc14RpcFeatureFlagENS_4lessIS3_EENS_9allocatorIS3_EEE16__construct_nodeIJRKS3_EEENS_10unique_ptrINS_11__tree_nodeIS3_PvEENS_22__tree_node_destructorINS6_ISF_EEEEEEDpOT_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2192 (libkrpc.so+0x90d18) #4 _ZNSt3__16__treeIN4kudu3rpc14RpcFeatureFlagENS_4lessIS3_EENS_9allocatorIS3_EEE15__emplace_multiIJRKS3_EEENS_15__tree_iteratorIS3_PNS_11__tree_nodeIS3_PvEElEEDpOT_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2240:25 (libkrpc.so+0x90c5a) #5 std::__1::__tree_iterator*, long> std::__1::__tree, std::__1::allocator >::__insert_multi(kudu::rpc::RpcFeatureFlag const&&&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1330:16 (libkrpc.so+0x9034c) #6 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1668 (libkrpc.so+0x9034c) #7 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #8 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #9 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #10 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #11 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #12 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #13 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #14 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #15 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #16 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0x9ed1d) #17 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1c8b3d) #18 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c4868) #19 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1cf7d9) #20 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1cf72a) #21 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1cf6b3) #22 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1cf4a9) #23 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #24 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) Thread T5 'client-negotiat' (tid=15200, finished) created by thread T3 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1bb9ef) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c6b85) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c3e72) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c263f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c40df) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c4021) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:582:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:552:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:336:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:841:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:310:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:471:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bbfc4) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ThreadSanitizer: reported 3 warnings