[SENTRY-1265] Sentry service should not require a TGT as it is not talking to other kerberos services as a client

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.8.0
• Assignee: Sravya Tirukkovalur
• Reporter: Sravya Tirukkovalur
• Created: 17/May/16 14:39
• Updated: 07/Jun/16 21:53
• Resolved: 20/May/16 21:27

Description

As part of renewThread we are logging out the subject and relogging in. This is causing a client request to fail if it happens in this logout -login window.

As only TGT needs renewal, we should never run the renewThread in Sentry given that Sentry never is a Kerberos Client to other Kerberos Services.
Stack trace from sentry server if a client requests while server is renewing:

2016-05-17 11:13:57,768 (pool-9-thread-2) [ERROR - org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:315)] SASL negotiation failure
javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)]
        at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:113)
        at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
        at javax.security.sasl.Sasl.createSaslServer(Sasl.java:509)
        at org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:140)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos credentails)
        at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:89)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:126)
        at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:192)
        at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:406)
        at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:60)
        at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:153)
        at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:96)
        ... 10 more
2016-05-17 11:13:57,769 (pool-9-thread-2) [ERROR - org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:296)] Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: Failure to initialize security context
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Failure to initialize security context
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 4 more
2016-05-17 11:13:57,769 (pool-9-thread-2) [DEBUG - org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:218)] failed to open server transport
org.apache.thrift.transport.TTransportException: Failure to initialize security context
        at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Stack trace from the client:

2016-05-17 11:13:57,769 (main) [DEBUG - org.apache.sentry.service.thrift.PoolClientInvocationHandler.invokeFromPool(PoolClientInvocationHandler.java:99)] Pool exception occured 
java.io.IOException: Transport exception while opening transport: Peer indicated failure: Failure to initialize security context
        at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.<init>(SentryPolicyServiceClientDefaultImpl.java:168)
        at org.apache.sentry.service.thrift.SentryServiceClientPoolFactory.create(SentryServiceClientPoolFactory.java:58)
        at org.apache.sentry.service.thrift.SentryServiceClientPoolFactory.create(SentryServiceClientPoolFactory.java:38)
        at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
        at org.apache.sentry.service.thrift.PoolClientInvocationHandler.invokeFromPool(PoolClientInvocationHandler.java:97)
        at org.apache.sentry.service.thrift.PoolClientInvocationHandler.invokeImpl(PoolClientInvocationHandler.java:70)
        at org.apache.sentry.service.thrift.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
        at com.sun.proxy.$Proxy7.listRoles(Unknown Source)
        at org.apache.sentry.service.thrift.SentryServiceIntegrationBase$1.runTestAsSubject(SentryServiceIntegrationBase.java:227)
        at org.apache.sentry.service.thrift.SentryServiceIntegrationBase$3.run(SentryServiceIntegrationBase.java:358)
        at org.apache.sentry.service.thrift.SentryServiceIntegrationBase$3.run(SentryServiceIntegrationBase.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.sentry.service.thrift.SentryServiceIntegrationBase.runTestAsSubject(SentryServiceIntegrationBase.java:355)
        at org.apache.sentry.service.thrift.SentryServiceIntegrationBase.after(SentryServiceIntegrationBase.java:223)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: org.apache.thrift.transport.TTransportException: Peer indicated failure: Failure to initialize security context
        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:199)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
        at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:130)
        at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.open(SentryPolicyServiceClientDefaultImpl.java:108)
        at org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.<init>(SentryPolicyServiceClientDefaultImpl.java:166)
        ... 43 more

Issue Links

• is duplicated by: SENTRY-662 SentryServiceIntegrationBase should use UGI based login
• relates to: SENTRY-428 Sentry service should periodically renew the server kerberos ticket
• links to: Review link