[ZOOKEEPER-3715] Kerberos Authentication related tests fail for new JDK versions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.0, 3.5.7
• Labels: pull-request-available
• Assignee: Mate Szalay-Beko
• Reporter: Mate Szalay-Beko
• Created: 03/Feb/20 13:00
• Updated: 14/Feb/20 15:23
• Resolved: 04/Feb/20 09:39

Description

using OpenJDK 1.8.242 or OpenJDK 11.0.6, I got some kerberos related exceptions when running the following, Kerberos Authentication related tests:
- QuorumKerberosAuthTest
- QuorumKerberosHostBasedAuthTest
- SaslKerberosAuthOverSSLTest
 
the error:

2020-02-03 12:11:07,197 [myid:localhost:11223] - ERROR [main-SendThread(localhost:11223):ZooKeeperSaslClient@336] - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: null (5001))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.
 

more detailed stack trace:

Found ticket for zkclient/[e-mail] to go to krbtgt/[e-mail] expiring on Tue Feb 04 13:49:14 CET 2020Found ticket for zkclient/[e-mail] to go to krbtgt/[e-mail] expiring on Tue Feb 04 13:49:14 CET 2020Entered Krb5Context.initSecContext with state=STATE_NEWService ticket not found in the subject>>> Credentials serviceCredsSingle: same realmUsing builtin default etypes for default_tgs_enctypesdefault etypes for default_tgs_enctypes: 18 17 16 23.>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes128CksumType>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType>>> KrbKdcReq send: kdc=localhost TCP:62653, timeout=30000, number of retries =3, #bytes=586>>> KDCCommunication: kdc=localhost TCP:62653, timeout=30000,Attempt =1, #bytes=586>>>DEBUG: TCPClient reading 112 bytes>>> KrbKdcReq send: #bytes read=112>>> KdcAccessibility: remove localhost:62653>>> KDCRep: init() encoding tag is 126 req type is 13>>>KRBError: sTime is Mon Feb 03 13:49:14 CET 2020 1580734154000 suSec is 100 error code is 5001 error Message is null crealm is EXAMPLE.COM sname is zkquorum/[e-mail] msgType is 30>>> Credentials serviceCredsSingle: same realmUsing builtin default etypes for default_tgs_enctypesdefault etypes for default_tgs_enctypes: 18 17 16 23.>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes128CksumType>>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType>>> KrbKdcReq send: kdc=localhost TCP:62653, timeout=30000, number of retries =3, #bytes=586>>> KDCCommunication: kdc=localhost TCP:62653, timeout=30000,Attempt =1, #bytes=586>>>DEBUG: TCPClient reading 112 bytes>>> KrbKdcReq send: #bytes read=112>>> KdcAccessibility: remove localhost:62653>>> KDCRep: init() encoding tag is 126 req type is 13>>>KRBError: sTime is Mon Feb 03 13:49:14 CET 2020 1580734154000 suSec is 100 error code is 5001 error Message is null crealm is EXAMPLE.COM sname is zkquorum/[e-mail] msgType is 30KrbException: null (5001) at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:70) at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226) at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237) at sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:400) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:287) at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:263) at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:118) at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:490) at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) at org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:320) at org.apache.zookeeper.client.ZooKeeperSaslClient$1.run(ZooKeeperSaslClient.java:317) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:317) at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:303) at org.apache.zookeeper.client.ZooKeeperSaslClient.sendSaslPacket(ZooKeeperSaslClient.java:366) at org.apache.zookeeper.client.ZooKeeperSaslClient.initialize(ZooKeeperSaslClient.java:403) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)Caused by: KrbException: Identifier doesn't match expected value (906) at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65) at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60) at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ... 20 more2020-02-03 13:49:14,942 [myid:localhost:11223] - ERROR [main-SendThread(localhost:11223):ZooKeeperSaslClient@336] - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: null (5001))]) occurred when evaluating Zookeeper Quorum Member's  received SASL token. Zookeeper Client will go to AUTH_FAILED state.

 
After trying this with different JDK versions, we see that the problem seems to appear

• between OpenJDK 8.232 and 8.242 for java 8
• and between 11.0.3 and 11.0.6 for java 11

There are a lot of kerberos related changes after 8.232: see https://hg.openjdk.java.net/jdk8u/jdk8u/jdk

 

Issue Links

• links to: GitHub Pull Request #1244
• links to: GitHub Pull Request #1248