Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 4.2.0
    • Fix Version/s: None
    • Component/s: core, security
    • Labels:
      None
    • Environment:

      Centos 6.7
      Hadoop 2.7.1
      Tested with java 1.7 & 1.8

    • Flags:
      Important

      Description

      Oozieserver can't connect to namenode, simply restarting oozie solve the mistake.
      The following exception is thrown at oozied kerberos ticket (oozie.service.HadoopAccessorService.kerberos.principal) expiration time (10h in my case) :

      2015-09-26 10:53:57,993 WARN RetryInvocationHandler:118 - SERVER[OOZIESERVER] Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing over NAMENODE/NAMENODE_IP:8020. Not retrying because failovers (15) exceeded maximum allowed (15)
      java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for oozie/OOZIESERVER@REALM to NAMENODE/NAMENODE_IP:8020; Host Details : local host is: "OOZIESERVER/OOZIESERVER_IP"; destination host is: "NAMENODE":8020;
      at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
      at org.apache.hadoop.ipc.Client.call(Client.java:1472)
      at org.apache.hadoop.ipc.Client.call(Client.java:1399)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
      at com.sun.proxy.$Proxy25.getListing(Unknown Source)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:554)
      at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      at com.sun.proxy.$Proxy26.getListing(Unknown Source)
      at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1969)
      at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1952)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:693)
      at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)
      at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)
      at org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)
      at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1485)
      at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1525)
      at org.apache.oozie.service.ShareLibService.getLatestLibPath(ShareLibService.java:687)
      at org.apache.oozie.service.ShareLibService$7.getValue(ShareLibService.java:742)
      at org.apache.oozie.service.ShareLibService$7.getValue(ShareLibService.java:737)
      at org.apache.oozie.util.Instrumentation.toString(Instrumentation.java:704)
      at org.apache.oozie.service.InstrumentationService$4.run(InstrumentationService.java:90)
      at org.apache.oozie.service.SchedulerService$2.run(SchedulerService.java:175)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Couldn't setup connection for oozie/OOZIESERVER@REALM to NAMENODE/NAMENODE_IP:8020
      at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:672)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
      at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
      at org.apache.hadoop.ipc.Client.call(Client.java:1438)
      ... 33 more
      Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
      at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
      at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
      at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)
      ... 36 more
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
      at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
      ... 45 more

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Alexandre LINTE LINTE
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: