Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5131 Known issues with KRPC (IMPALA-2567)
  3. IMPALA-5596

Data load failed with " Failed to find any Kerberos tgt" on secure cluster

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 2.11.0
    • Component/s: Distributed Exec
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      Error from node which failed to scan the data

      W0627 23:31:48.568421 31308 UserGroupInformation.java:1920] PriviledgedActionException as:impala/va1045.foo@foo.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      W0627 23:31:48.575223 31308 Client.java:709] Couldn't setup connection for impala/va1045.foo@foo.COM to vc1512.foo/10.:8020
      Java exception follows:
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
      	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
      	at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1478)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
      	at com.sun.proxy.$Proxy19.getBlockLocations(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
      	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
      	at com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1282)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1269)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1257)
      	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:305)
      	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:271)
      	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:263)
      	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1588)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:331)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:331)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 33 more
      W0627 23:31:48.576414 31308 UserGroupInformation.java:1920] PriviledgedActionException as:impala/va1045.foo@foo.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for impala/va1045.foo@foo.COM to vc1512.foo/10.:8020
      I0627 23:31:48.603538 31308 status.cc:123] Failed to open HDFS file hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
      Error(255): Unknown error 255
      Root cause: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
          @           0x90cf99  impala::Status::Status()
          @           0xbb6434  impala::DiskIoMgr::ScanRange::Open()
          @           0xbaf00f  impala::DiskIoMgr::ReadRange()
          @           0xbb0234  impala::DiskIoMgr::WorkLoop()
          @           0xd75742  impala::Thread::SuperviseThread()
          @           0xd75fe4  boost::detail::thread_data<>::run()
          @          0x102f32a  thread_proxy
          @       0x3434607aa1  (unknown)
          @       0x34342e893d  (unknown)
      I0627 23:31:48.606230 171088 hdfs-scan-node.cc:522] Error preparing scanner for scan range hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq(264400819:102400).
      I0627 23:31:50.615875 171066 query-state.cc:327] Instance completed. instance_id=3b4aeb6665ce8dae:7757b04300000003 #in-flight=1 status=GENERAL: Failed to open HDFS file hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
      Error(255): Unknown error 255
      Root cause: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      I0627 23:31:50.616274 171066 query-state.cc:337] Cancel: query_id=3b4aeb6665ce8dae:7757b04300000000
      

      Error from coordinator node

      W0627 23:32:02.232866 191757 UserGroupInformation.java:1920] PriviledgedActionException as:impala/va1026.foo.com@foo.COM (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      W0627 23:32:02.237445 191757 Client.java:709] Couldn't setup connection for impala/va1026.foo.com@foo.COM to vc1512.foo.com/10.:8020
      Java exception follows:
      javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
      	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
      	at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1478)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
      	at com.sun.proxy.$Proxy19.delete(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
      	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
      	at com.sun.proxy.$Proxy20.delete(Unknown Source)
      	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2062)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:684)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:680)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:680)
      Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
      	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
      	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
      	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
      	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
      	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
      	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
      	... 27 more
      W0627 23:32:02.238351 191757 UserGroupInformation.java:1920] PriviledgedActionException as:impala/va1026.foo.com@foo.COM (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for impala/va1026.foo.com@foo.COM to vc1512.foo.com/10.:8020
      I0627 23:32:02.939185 45977 impala-beeswax-server.cc:233] close(): query_id=3b4aeb6665ce8dae:7757b04300000000
      I0627 23:32:02.939621 45977 impala-server.cc:952] UnregisterQuery(): query_id=3b4aeb6665ce8dae:7757b04300000000
      I0627 23:32:02.939944 45977 impala-server.cc:1037] Cancel(): query_id=3b4aeb6665ce8dae:7757b04300000000
      I0627 23:32:02.971863 45977 query-exec-mgr.cc:138] ReleaseQueryState(): query_id=3b4aeb6665ce8dae:7757b04300000000 refcnt=1
      I0627 23:32:03.003536 45977 impala-server.cc:1713] Connection from client 10.17.187.36:48502 closed, closing 1 associated session(s)
      I0627 23:32:03.011557 45977 status.cc:123] Session closed
          @           0x90cf99  impala::Status::Status()
          @           0xc4421a  impala::ImpalaServer::CloseSessionInternal()
          @           0xc44900  impala::ImpalaServer::ConnectionEnd()
          @           0xb39640  impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
          @          0x1d24d73  apache::thrift::server::TThreadPoolServer::Task::run()
          @          0x1d0be59  apache::thrift::concurrency::ThreadManager::Worker::run()
          @           0xb3f119  impala::ThriftThread::RunRunnable()
          @           0xb3fb72  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @           0xd75742  impala::Thread::SuperviseThread()
          @           0xd75fe4  boost::detail::thread_data<>::run()
          @          0x102f32a  thread_proxy
          @       0x37dd207aa1  (unknown)
          @       0x37dcee893d  (unknown)
      W0627 23:32:12.421926 44973 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1498631482824
      W0627 23:32:12.422919 44973 UserGroupInformation.java:1027] No TGT after renewal. Aborting renew thread for impala/va1026.foo.com@foo.COM
      I0627 23:43:13.903360 44895 init.cc:294] Successfully renewed kerberos TGT
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sailesh Sailesh Mukil
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: