[ZOOKEEPER-3576] Zookeeper Fails with AUTH_FAILED state with SASL

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.10
• Component/s: kerberos, security
• Assignee: Unassigned
• Reporter: Ahshan
• Created: 14/Oct/19 06:02
• Updated: 14/Oct/19 07:44

Description

Although i'm able to authenticate successfully with the kerberoes account "zookeeper/[e-mail]" , i still happen to encounter  AUTH_FAILED during client Authentication

Following is the verification made from my end :

1. Checked DNS ( Both Forward and Backward)

nslookup kafka-d1.eng.company.com
Server: 172.16.2.3
Address: 172.16.2.3#53

Name: kafka-d1.eng.company.com
Address: 10.14.61.17

Reverse DNS

nslookup 10.14.61.17
Server: 172.16.2.3
Address: 172.16.2.3#53

17.61.14.10.in-addr.arpa name = kafka-d1.eng.company.com.

 

2. Kerberoes Authentication

kinit -kt /etc/keytabs/zookeeper.keytab -V zookeeper/kafka-d1.eng.company.com
Using default cache: /tmp/krb5cc_0
Using principal: zookeeper/[e-mail]
Using keytab: /etc/keytabs/zookeeper.keytab
Authenticated to Kerberos v5

 

Below is the krb5 configuration File:

cat /etc/krb5.conf
[libdefaults]
default_realm = COMPANY.COM
dns_lookup_kdc = true
dns_lookup_realm = true
ticket_lifetime = 86400
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = aes256-cts
default_tkt_enctypes = aes256-cts
permitted_enctypes = aes256-cts
udp_preference_limit = 1
kdc_timeout = 3000
ignore_acceptor_hostname = true
[realms]
COMPANY.COM =

{ kdc = srv-ussc-dc01e.company.com admin_server = srv-exxx.company.com kdc = srv-exxxe.company.com }

[domain_realm]
kafka-d1.eng.company.com = COMPANY.COM

 

export JVMFLAGS=-Djava.security.auth.login.config=/usr/share/zookeeper/conf/client_jaas.conf -Dsun.security.krb5.debug=true

 

cat /usr/share/zookeeper/conf/client_jaas.conf
Client

{ com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true debug=true keyTab="/etc/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/[e-mail]; }

;

Error Message :zoo.cfgzookeeper_server.log

./zkCli.sh -server kafka-d1.eng.company.com:2181
Connecting to kafka-d1.eng.company.com:2181
2019-10-14 02:08:16,625 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-10-14 02:08:16,628 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=kafka-d1.eng.company.com
2019-10-14 02:08:16,628 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/share/zookeeper/bin/../build/classes:/usr/share/zookeeper/bin/../build/lib/*.jar:/usr/share/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/share/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/share/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/share/zookeeper/bin/../zookeeper-3.4.10.jar:/usr/share/zookeeper/bin/../src/java/lib/*.jar:/usr/share/zookeeper/bin/../conf:
2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/usr/share/zookeeper-3.4.10/bin
2019-10-14 02:08:16,632 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-d1.eng.company.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
Welcome to ZooKeeper!
JLine support is enabled
Debug is  true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/[e-mail] tryFirstPass is false useFirstPass is false storePass is false clearPass is false
[zk: kafka-d1.eng.company.com:2181(CONNECTING) 0] principal is zookeeper/[e-mail]
Will use keytab
Commit Succeeded 2019-10-14 02:08:16,971 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):Login@295] - Client successfully logged in.
2019-10-14 02:08:16,973 [myid:] - INFO  [Thread-1:Login$1@128] - TGT refresh thread started.
2019-10-14 02:08:16,975 [myid:] - INFO  [Thread-1:Login@303] - TGT valid starting at:        Mon Oct 14 02:08:16 EDT 2019
2019-10-14 02:08:16,976 [myid:] - INFO  [Thread-1:Login@304] - TGT expires:                  Mon Oct 14 12:08:16 EDT 2019
2019-10-14 02:08:16,976 [myid:] - INFO  [Thread-1:Login$1@183] - TGT refresh sleeping until: Mon Oct 14 10:08:57 EDT 2019
2019-10-14 02:08:16,977 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):SecurityUtils$1@124] - Client will use GSSAPI as SASL mechanism.
2019-10-14 02:08:16,988 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server kafka-d1.eng.company.com/10.14.61.17:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2019-10-14 02:08:16,994 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@876] - Socket connection established to kafka-d1.eng.company.com/10.14.61.17:2181, initiating session
2019-10-14 02:08:17,002 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server kafka-d1.eng.company.com/10.14.61.17:2181, sessionid = 0x16dc8cbdb3b0002, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null
2019-10-14 02:08:17,024 [myid:] - ERROR [main-SendThread(kafka-d1.eng.company.com:2181):ZooKeeperSaslClient@247] - SASL authentication failed using login context 'Client'.WATCHER::WatchedEvent state:AuthFailed type:None path:null