Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1907

Shell Interpreter does not renew ticket on secure cluster

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Kerberos ticket and renew lifetime are set to 1 hour. On accessing secure hadoop from shell interpreter, it does kinit and returns result successfully but after 1 hour, the ticket gets expired and hadoop list fails with below exception.

      %sh
      hadoop fs -ls /
      
      17/01/05 09:29:45 WARN ipc.Client: Exception encountered while connecting to the server : 
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
      	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
      	at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1449)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1396)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
      	at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
      	at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
      	at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
      	at org.apache.hadoop.fs.Globber.glob(Globber.java:265)
      	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
      	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
      	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
      	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
      	at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
      	at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
      	at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      	at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
      	... 41 more
      17/01/05 09:29:45 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying because try once and fail.
      java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020; 
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
      	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1496)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1396)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
      	at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:816)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
      	at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2158)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1423)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1419)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1419)
      	at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
      	at org.apache.hadoop.fs.Globber.glob(Globber.java:265)
      	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1674)
      	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:326)
      	at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
      	at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
      	at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
      	at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
      	at org.apache.hadoop.fs.FsShell.run(FsShell.java:297)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      	at org.apache.hadoop.fs.FsShell.main(FsShell.java:350)
      Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:720)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
      	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
      	at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1449)
      	... 29 more
      Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
      	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
      	... 32 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
      	... 41 more
      ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "zeppelin1.hwxblr.com/10.0.1.57"; destination host is: "zeppelin1.hwxblr.com":8020; 
      ExitValue: 1
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                prabhjyotsingh Prabhjyot Singh
                Reporter:
                Prabhu Joseph Prabhu Joseph
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: