Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1596

Multiple Subject sharing Kerberos TGT - causes services to fail

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 1.0.0, 0.10.1, 2.0.0
    • Fix Version/s: 1.0.0, 0.10.1, 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      With multiple threads accessing same Subject, it can cause ServiceTicket in use be by one thread be destroyed by another thread.

      Running BasicDRPCTopology with high parallelism in secure cluster would reproduce the issue.

      Here is sample log from such a scenarios:

      2016-01-20 15:52:26.904 o.a.t.t.TSaslTransport [ERROR] SASL negotiation failure
      javax.security.sasl.SaslException: GSS initiate failed
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_40]
              at org.apache.thrift7.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[storm-core-0.10.1.y.jar:0.10.1.y]
              at org.apache.thrift7.transport.TSaslTransport.open(TSaslTransport.java:271) [storm-core-0.10.1.y.jar:0.10.1.y]
              at org.apache.thrift7.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:195) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin$1.run(KerberosSaslTransportPlugin.java:191) [storm-core-0.10.1.y.jar:0.10.1.y]
              at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_40]
              at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_40]
              at backtype.storm.security.auth.kerberos.KerberosSaslTransportPlugin.connect(KerberosSaslTransportPlugin.java:190) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:54) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:109) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.drpc.DRPCInvocationsClient.reconnectClient(DRPCInvocationsClient.java:57) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.drpc.ReturnResults.reconnectClient(ReturnResults.java:113) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.drpc.ReturnResults.execute(ReturnResults.java:103) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.daemon.executor$fn__6377$tuple_action_fn__6379.invoke(executor.clj:689) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.daemon.executor$mk_task_receiver$fn__6301.invoke(executor.clj:448) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.disruptor$clojure_handler$reify__6018.onEvent(disruptor.clj:40) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:437) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:416) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.daemon.executor$fn__6377$fn__6390$fn__6441.invoke(executor.clj:801) [storm-core-0.10.1.y.jar:0.10.1.y]
              at backtype.storm.util$async_loop$fn__742.invoke(util.clj:482) [storm-core-0.10.1.y.jar:0.10.1.y]
              at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
              at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
      Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: The ticket isn't for us (35) - BAD TGS SERVER NAME)
              at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
              ... 23 more
      Caused by: sun.security.krb5.KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
              at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73) ~[?:1.8.0_40]
              at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259) ~[?:1.8.0_40]
              at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270) ~[?:1.8.0_40]
              at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302) ~[?:1.8.0_40]
              at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120) ~[?:1.8.0_40]
              at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_40]
              at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
              ... 23 more
      Caused by: sun.security.krb5.Asn1Exception: Identifier doesn't match expected value (906)
              at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140) ~[?:1.8.0_40]
              at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65) ~[?:1.8.0_40]
              at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60) ~[?:1.8.0_40]
              at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55) ~[?:1.8.0_40]
              at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259) ~[?:1.8.0_40]
              at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270) ~[?:1.8.0_40]
              at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302) ~[?:1.8.0_40]
              at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120) ~[?:1.8.0_40]
              at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458) ~[?:1.8.0_40]
              at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248) ~[?:1.8.0_40]
              at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_40]
              at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_40]
              ... 23 more
      
      
      

        Attachments

          Activity

            People

            • Assignee:
              kishorvpatil Kishor Patil
              Reporter:
              kishorvpatil Kishor Patil
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: