Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-3576

Zookeeper Fails with AUTH_FAILED state with SASL

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.4.10
    • Fix Version/s: None
    • Component/s: kerberos, security
    • Labels:
      None

      Description

      Although i'm able to authenticate successfully with the kerberoes account "zookeeper/kafka-d1.eng.company.com@COMPANY.COM" , i still happen to encounter  AUTH_FAILED during client Authentication

      Following is the verification made from my end :

      1. Checked DNS ( Both Forward and Backward)

      nslookup kafka-d1.eng.company.com
      Server: 172.16.2.3
      Address: 172.16.2.3#53

      Name: kafka-d1.eng.company.com
      Address: 10.14.61.17

      Reverse DNS

      nslookup 10.14.61.17
      Server: 172.16.2.3
      Address: 172.16.2.3#53

      17.61.14.10.in-addr.arpa name = kafka-d1.eng.company.com.

       

      2. Kerberoes Authentication

      kinit -kt /etc/keytabs/zookeeper.keytab -V zookeeper/kafka-d1.eng.company.com
      Using default cache: /tmp/krb5cc_0
      Using principal: zookeeper/kafka-d1.eng.company.com@COMPANY.COM
      Using keytab: /etc/keytabs/zookeeper.keytab
      Authenticated to Kerberos v5

       

      Below is the krb5 configuration File:

      cat /etc/krb5.conf
      [libdefaults]
      default_realm = COMPANY.COM
      dns_lookup_kdc = true
      dns_lookup_realm = true
      ticket_lifetime = 86400
      renew_lifetime = 604800
      forwardable = true
      default_tgs_enctypes = aes256-cts
      default_tkt_enctypes = aes256-cts
      permitted_enctypes = aes256-cts
      udp_preference_limit = 1
      kdc_timeout = 3000
      ignore_acceptor_hostname = true
      [realms]
      COMPANY.COM =

      { kdc = srv-ussc-dc01e.company.com admin_server = srv-exxx.company.com kdc = srv-exxxe.company.com }

      [domain_realm]
      kafka-d1.eng.company.com = COMPANY.COM

       

      export JVMFLAGS=-Djava.security.auth.login.config=/usr/share/zookeeper/conf/client_jaas.conf -Dsun.security.krb5.debug=true

       

      cat /usr/share/zookeeper/conf/client_jaas.conf
      Client

      { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true debug=true keyTab="/etc/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/kafka-d1.eng.company.com@COMPANY.COM; }

      ;

      Error Message :zoo.cfgzookeeper_server.log

      ./zkCli.sh -server kafka-d1.eng.company.com:2181
      Connecting to kafka-d1.eng.company.com:2181
      2019-10-14 02:08:16,625 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
      2019-10-14 02:08:16,628 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=kafka-d1.eng.company.com
      2019-10-14 02:08:16,628 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_201
      2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
      2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/jdk1.8.0_201/jre
      2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/usr/share/zookeeper/bin/../build/classes:/usr/share/zookeeper/bin/../build/lib/*.jar:/usr/share/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/share/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/share/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/share/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/share/zookeeper/bin/../lib/jline-0.9.94.jar:/usr/share/zookeeper/bin/../zookeeper-3.4.10.jar:/usr/share/zookeeper/bin/../src/java/lib/*.jar:/usr/share/zookeeper/bin/../conf:
      2019-10-14 02:08:16,630 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-327.el7.x86_64
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
      2019-10-14 02:08:16,631 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/usr/share/zookeeper-3.4.10/bin
      2019-10-14 02:08:16,632 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=kafka-d1.eng.company.com:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@306a30c7
      Welcome to ZooKeeper!
      JLine support is enabled
      Debug is  true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM tryFirstPass is false useFirstPass is false storePass is false clearPass is false
      [zk: kafka-d1.eng.company.com:2181(CONNECTING) 0] principal is zookeeper/kafka-d1.eng.company.com@COMPANY.COM
      Will use keytab
      Commit Succeeded 2019-10-14 02:08:16,971 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):Login@295] - Client successfully logged in.
      2019-10-14 02:08:16,973 [myid:] - INFO  [Thread-1:Login$1@128] - TGT refresh thread started.
      2019-10-14 02:08:16,975 [myid:] - INFO  [Thread-1:Login@303] - TGT valid starting at:        Mon Oct 14 02:08:16 EDT 2019
      2019-10-14 02:08:16,976 [myid:] - INFO  [Thread-1:Login@304] - TGT expires:                  Mon Oct 14 12:08:16 EDT 2019
      2019-10-14 02:08:16,976 [myid:] - INFO  [Thread-1:Login$1@183] - TGT refresh sleeping until: Mon Oct 14 10:08:57 EDT 2019
      2019-10-14 02:08:16,977 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):SecurityUtils$1@124] - Client will use GSSAPI as SASL mechanism.
      2019-10-14 02:08:16,988 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server kafka-d1.eng.company.com/10.14.61.17:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
      2019-10-14 02:08:16,994 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@876] - Socket connection established to kafka-d1.eng.company.com/10.14.61.17:2181, initiating session
      2019-10-14 02:08:17,002 [myid:] - INFO  [main-SendThread(kafka-d1.eng.company.com:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server kafka-d1.eng.company.com/10.14.61.17:2181, sessionid = 0x16dc8cbdb3b0002, negotiated timeout = 30000WATCHER::WatchedEvent state:SyncConnected type:None path:null
      2019-10-14 02:08:17,024 [myid:] - ERROR [main-SendThread(kafka-d1.eng.company.com:2181):ZooKeeperSaslClient@247] - SASL authentication failed using login context 'Client'.WATCHER::WatchedEvent state:AuthFailed type:None path:null

       

       

        Attachments

        1. zoo.cfg
          1 kB
          Ahshan
        2. zookeeper_server.log
          4 kB
          Ahshan

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ahshan.md@gmail.com Ahshan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: