[ZEPPELIN-3356] Zeppelin FileSystemStorage reloginFromKeytab needed

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.0
• Fix Version/s: 0.8.0
• Assignee: Greg Senia
• Reporter: Greg Senia
• Created: 21/Mar/18 16:22
• Updated: 18/Apr/18 03:39
• Resolved: 18/Apr/18 03:38

Description

After running the current Zeppelin snapshot. Users were complaining that they could no longer make notebook changes as we are storing the notebook configuration and settings in HDFS. When I went to look at the Zeppelin logs I saw the following below. I propose some slight changes as similar code is used in KMSClientProvider in core Hadoop and I have used similar code in homegrown HDFS management software. This error occurs exactly 7days after the initial keytab is logged in at startup. Unfortunately getCurrentUser doesn't seem to relogin a keytab once the ticket reaches max expiration in our case 7days.

 

 

INFO [2018-03-16 08:50:01,275] ({qtp2081303229-1297} RetryInvocationHandler.java[invoke]:144) - Exception while invoking create of class ClientNamenodeProtocolTranslatorPB over ha20p01nn.pro
d.hdp.example.com/10.70.1.6:8020. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism l
evel: Failed to find any Kerberos tgt)]; Host Details : local host is: "ha21p05en.prod.hdp.example.com/10.70.8.211"; destination host is: "ha20p01nn.prod.hdp.example.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy24.create(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy25.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at org.apache.zeppelin.notebook.FileSystemStorage$6.call(FileSystemStorage.java:144)
at org.apache.zeppelin.notebook.FileSystemStorage$6.call(FileSystemStorage.java:138)
at org.apache.zeppelin.notebook.FileSystemStorage$7.run(FileSystemStorage.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.zeppelin.notebook.FileSystemStorage.callHdfsOperation(FileSystemStorage.java:159)
at org.apache.zeppelin.notebook.FileSystemStorage.writeFile(FileSystemStorage.java:138)
at org.apache.zeppelin.storage.FileSystemConfigStorage.save(FileSystemConfigStorage.java:66)
at org.apache.zeppelin.interpreter.InterpreterSettingManager.saveToFile(InterpreterSettingManager.java:285)
at org.apache.zeppelin.interpreter.InterpreterSettingManager.setInterpreterBinding(InterpreterSettingManager.java:705)
at org.apache.zeppelin.notebook.Notebook.bindInterpretersToNote(Notebook.java:275)
at org.apache.zeppelin.notebook.Notebook.createNote(Notebook.java:166)
at org.apache.zeppelin.socket.NotebookServer.createNote(NotebookServer.java:1037)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:226)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:687)

Issue Links

• is related to: ZEPPELIN-2385 Release 0.8.0
• links to: GitHub Pull Request #2886
• links to: GitHub Pull Request #2924