Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3668

Unable to authenticate Kafka broker to secured Zookeeper

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: 0.9.0.0, 0.9.0.1
    • Fix Version/s: 0.9.0.0, 0.9.0.1
    • Component/s: None
    • Labels:
      None
    • Environment:
      Red Hat Enterprise Linux Server release 7.0 (Maipo)
      Java 1.8.0_66-b17
      Kafka 0.9.0.0 and 0.9.0.1
    • Flags:
      Important

      Description

      Hello,

      we are running into trouble when trying to connect Kafka broker to secured Zookeeper, Kerberos protected.
      Configuration is as simple as possible: 1 Zookeeper, 1 Kafka broker and Kerberos. All running on local machine.

      Zookeeper successfully starts and receives TGT from Kerberos AS_REQ. Then Kafka broker obtains TGT from AS_REQ, but it is unable to get TGS from TGS_REQ because <unknown server> as krb5kdc.log shows:
      krb5kdc.log
      ...
      May 06 17:41:42 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes

      {18 17 16 23}) 10.116.93.88: ISSUE: authtime 1462545702, etypes {rep=18 tkt=18 ses=18}, zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
      May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): AS_REQ (4 etypes {18 17 16 23}

      ) 10.116.93.88: ISSUE: authtime 1462545864, etypes

      {rep=18 tkt=18 ses=18}

      , kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for krbtgt/CA.SBRF.RU@CA.SBRF.RU
      May 06 17:44:24 SBT-IPO-204.ca.sbrf.ru krb5kdc[1580](info): TGS_REQ (4 etypes

      {18 17 16 23}

      ) 10.116.93.88: LOOKING_UP_SERVER: authtime 0, kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU for <unknown server>, Server not found in Kerberos database

      What is the possible reason of this problem?

      KAFKA CONFIG:

      zookeeper.properties
      dataDir=/tmp/zookeeper
      clientPort=2181
      authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
      jaasLoginRenew=3600000

      server.properties
      broker.id=0
      log.dirs=/tmp/kafka-logs
      listeners=SASL_PLAINTEXT://10.116.93.88:9092
      security.inter.broker.protocol=SASL_PLAINTEXT
      zookeeper.connect=10.116.93.88:2181
      sasl.kerberos.service.name=kafka
      authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
      zookeeper.set.acl=true
      #allow.everyone.if.no.acl.found=true
      #sasl.enabled.mechanisms=GSSAPI
      #sasl.mechanism.inter.broker.protocol=GSSAPI

      JVM params:

      Kafka:
      -Djava.security.krb5.conf=/etc/krb5.conf
      -Djava.security.auth.login.config=config/kafka-broker-jaas.conf

      Zookeeper:
      -Djava.security.krb5.conf=/etc/krb5.conf
      -Djava.security.auth.login.config=config/zookeeper.conf

      JAAS files:

      kafka-broker-jaas.conf:
      KafkaServer

      { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.keytab" debug=true useTicketCache=false principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU"; };
      Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/security/keytabs/kafka.keytab" debug=true useTicketCache=false principal="kafka/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU"; }

      ;

      zookeeper-jaas.conf
      Server

      { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false debug=true principal="zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU"; }

      ;

      KERBEROS 5 CONFIG:

      krb5.conf
      [logging]
      default = FILE:/var/log/krb5libs.log
      kdc = FILE:/var/log/krb5kdc.log
      admin_server = FILE:/var/log/kadmind.log

      [libdefaults]
      dns_lookup_realm = false
      ticket_lifetime = 24h
      renew_lifetime = 7d
      forwardable = true
      rdns = false
      default_realm = CA.SBRF.RU
      default_ccache_name = KEYRING:persistent:%

      {uid}

      [realms]
      CA.SBRF.RU =

      { kdc = SBT-IPO-204.ca.sbrf.ru admin_server = SBT-IPO-204.ca.sbrf.ru }

      [domain_realm]
      .ca.sbrf.ru = CA.SBRF.RU
      ca.sbrf.ru = CA.SBRF.RU

      kdc.conf
      [kdcdefaults]
      kdc_ports = 88
      kdc_tcp_ports = 88

      [realms]
      CA.SBRF.RU =

      { #master_key_type = aes256-cts acl_file = /var/kerberos/krb5kdc/kadm5.acl dict_file = /usr/share/dict/words admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal }

      kadm.conf
      */admin@CA.SBRF.RU *

      LOGS:

      Zookeeper: bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

      ...
      [2016-05-06 17:41:42,750] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
      [2016-05-06 17:41:42,750] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
      Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /etc/security/keytabs/zookeeper.keytab refreshKrb5Config is false principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU tryFirstPass is false useFirstPass is false storePass is false clearPass is false
      principal is zookeeper/SBT-IPO-204.ca.sbrf.ru@CA.SBRF.RU
      Will use keytab
      Commit Succeeded

      [2016-05-06 17:41:43,137] INFO successfully logged in. (org.apache.zookeeper.Login)
      [2016-05-06 17:41:43,143] INFO TGT refresh thread started. (org.apache.zookeeper.Login)
      [2016-05-06 17:41:43,150] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
      [2016-05-06 17:41:43,169] INFO TGT valid starting at: Fri May 06 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
      [2016-05-06 17:41:43,170] INFO TGT expires: Sat May 07 17:41:42 MSK 2016 (org.apache.zookeeper.Login)
      [2016-05-06 17:41:43,170] INFO TGT refresh sleeping until: Sat May 07 14:04:31 MSK 2016 (org.apache.zookeeper.Login)

      ...Here Kafka starts...

      [2016-05-06 17:44:24,933] INFO Accepted socket connection from /10.116.93.88:58825 (org.apache.zookeeper.server.NIOServerCnxnFactory)
      [2016-05-06 17:44:24,952] ERROR Zookeeper Server failed to create a SaslServer to interact with a client during session initiation: javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)] (org.apache.zookeeper.server.ZooKeeperSaslServer)
      javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
      at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(Unknown Source)
      at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(Unknown Source)
      at javax.security.sasl.Sasl.createSaslServer(Unknown Source)
      at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:118)
      at org.apache.zookeeper.server.ZooKeeperSaslServer$1.run(ZooKeeperSaslServer.java:114)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Unknown Source)
      at org.apache.zookeeper.server.ZooKeeperSaslServer.createSaslServer(ZooKeeperSaslServer.java:114)
      at org.apache.zookeeper.server.ZooKeeperSaslServer.<init>(ZooKeeperSaslServer.java:48)
      at org.apache.zookeeper.server.NIOServerCnxn.<init>(NIOServerCnxn.java:100)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.createConnection(NIOServerCnxnFactory.java:161)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:202)
      at java.lang.Thread.run(Unknown Source)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
      at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Unknown Source)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
      at sun.security.jgss.GSSManagerImpl.getCredentialElement(Unknown Source)
      at sun.security.jgss.GSSCredentialImpl.add(Unknown Source)
      at sun.security.jgss.GSSCredentialImpl.<init>(Unknown Source)
      at sun.security.jgss.GSSManagerImpl.createCredential(Unknown Source)
      ... 13 more
      [2016-05-06 17:44:24,961] INFO Client attempting to establish new session at /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
      [2016-05-06 17:44:24,963] INFO Creating new log file: log.53 (org.apache.zookeeper.server.persistence.FileTxnLog)
      [2016-05-06 17:44:24,972] INFO Established session 0x154868461350000 with negotiated timeout 6000 for client /10.116.93.88:58825 (org.apache.zookeeper.server.ZooKeeperServer)
      [2016-05-06 17:44:28,997] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
      EndOfStreamException: Unable to read additional data from client sessionid 0x154868461350000, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
      at java.lang.Thread.run(Unknown Source)
      [2016-05-06 17:44:29,001] INFO Closed socket connection for client /10.116.93.88:58825 which had sessionid 0x154868461350000 (org.apache.zookeeper.server.NIOServerCnxn)
      [2016-05-06 17:44:33,001] INFO Expiring session 0x154868461350000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
      [2016-05-06 17:44:33,002] INFO Processed session termination for sessionid: 0x154868461350000 (org.apache.zookeeper.server.PrepRequestProcessor)

      Kafka: bin/kafka-server-start.sh -daemon config/server.properties

      ...
      [2016-05-06 17:44:24,353] INFO starting (kafka.server.KafkaServer)
      [2016-05-06 17:44:24,360] INFO Connecting to zookeeper on 10.116.93.88:2181 (kafka.server.KafkaServer)
      [2016-05-06 17:44:30,428] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
      org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
      at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
      at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
      at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
      at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
      at kafka.Kafka$.main(Kafka.scala:67)
      at kafka.Kafka.main(Kafka.scala)
      [2016-05-06 17:44:30,431] INFO shutting down (kafka.server.KafkaServer)
      [2016-05-06 17:44:30,438] INFO shut down completed (kafka.server.KafkaServer)
      [2016-05-06 17:44:30,439] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
      org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
      at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
      at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
      at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
      at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
      at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
      at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
      at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
      at kafka.Kafka$.main(Kafka.scala:67)
      at kafka.Kafka.main(Kafka.scala)
      [2016-05-06 17:44:30,442] INFO shutting down (kafka.server.KafkaServer)

      UPDATE:

      This is not actually a Kafka issue.
      The problem was at specifying the wrong FQDN (Fully Qualified Domain Name) at DNS.

      Kafka box has two DNS records:

      • with uppercase
      • with lowercase

      Kafka requests user with lowercase FQDN.

      Example:
      SBT-IPO-204.ca.sbrf.ru
      should be
      sbt-ipo-204.ca.sbrf.ru in JAAS file.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alex.dunayevsky Alex Dunayevsky
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: