Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
0.10.0, 1.0.0, 0.10.1, 2.0.0
-
None
-
None
-
Important
Description
With multiple threads accessing same Subject, it can cause ServiceTicket in use be by one thread be destroyed by another thread.
Running BasicDRPCTopology with high parallelism in secure cluster would reproduce the issue.
Here is sample log from such a scenarios:
2016-01-20 15:52:26.904 o.a.t.t.TSaslTransport [ERROR] SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_40]
at org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[storm-core-0.10.1.y.jar:0.10.1.y]
at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271) [storm-core-0.10.1.y.jar:0.10.1.y]
at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:195) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:191) [storm-core-0.10.1.y.jar:0.10.1.y]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_40]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_40]
at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:190) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:54) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:109) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.drpc.DRPCInvocationsClient.reconnectClient(DRPCInvocationsClient.java:57) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.drpc.ReturnResults.reconnectClient(ReturnResults.java:113) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.drpc.ReturnResults.execute(ReturnResults.java:103) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.daemon.executor$fn__6377$tuple_action_fn__6379.invoke(executor.clj:689) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.daemon.executor$mk_task_receiver$fn__6301.invoke(executor.clj:448) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.disruptor$clojure_handler$reify__6018.onEvent(disruptor.clj:40) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:437) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:416) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.daemon.executor$fn__6377$fn__6390$fn__6441.invoke(executor.clj:801) [storm-core-0.10.1.y.jar:0.10.1.y]
at backtype.storm.util$async_loop$fn__742.invoke(util.clj:482) [storm-core-0.10.1.y.jar:0.10.1.y]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: The ticket isn't for us (35) - BAD TGS SERVER NAME)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
... 23 more
Caused by: sun.security.krb5.KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270) ~[?:1.8.0_40]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302) ~[?:1.8.0_40]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120) ~[?:1.8.0_40]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_40]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
... 23 more
Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) ~[?:1.8.0_40]
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65) ~[?:1.8.0_40]
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259) ~[?:1.8.0_40]
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270) ~[?:1.8.0_40]
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302) ~[?:1.8.0_40]
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120) ~[?:1.8.0_40]
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_40]
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
... 23 more