[IMPALA-5596] Data load failed with " Failed to find any Kerberos tgt" on secure cluster

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.11.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-7
• Assignee: Sailesh Mukil
• Reporter: Mostafa Mokhtar
• Created: 28/Jun/17 19:15
• Updated: 24/Oct/17 23:47
• Resolved: 24/Oct/17 23:45

Description

Error from node which failed to scan the data

W0627 23:31:48.568421 31308 UserGroupInformation.java:1920] PriviledgedActionException as:impala/[e-mail] (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0627 23:31:48.575223 31308 Client.java:709] Couldn't setup connection for impala/[e-mail] to vc1512.foo/10.:8020
Java exception follows:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
	at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
	at org.apache.hadoop.ipc.Client.call(Client.java:1478)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy19.getBlockLocations(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:256)
	at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy20.getBlockLocations(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1282)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1269)
	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1257)
	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:305)
	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:271)
	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:263)
	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1588)
	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:335)
	at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:331)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:331)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
	... 33 more
W0627 23:31:48.576414 31308 UserGroupInformation.java:1920] PriviledgedActionException as:impala/[e-mail] (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for impala/[e-mail] to vc1512.foo/10.:8020
I0627 23:31:48.603538 31308 status.cc:123] Failed to open HDFS file hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
Error(255): Unknown error 255
Root cause: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    @           0x90cf99  impala::Status::Status()
    @           0xbb6434  impala::DiskIoMgr::ScanRange::Open()
    @           0xbaf00f  impala::DiskIoMgr::ReadRange()
    @           0xbb0234  impala::DiskIoMgr::WorkLoop()
    @           0xd75742  impala::Thread::SuperviseThread()
    @           0xd75fe4  boost::detail::thread_data<>::run()
    @          0x102f32a  thread_proxy
    @       0x3434607aa1  (unknown)
    @       0x34342e893d  (unknown)
I0627 23:31:48.606230 171088 hdfs-scan-node.cc:522] Error preparing scanner for scan range hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq(264400819:102400).
I0627 23:31:50.615875 171066 query-state.cc:327] Instance completed. instance_id=3b4aeb6665ce8dae:7757b04300000003 #in-flight=1 status=GENERAL: Failed to open HDFS file hdfs://vc1512.foo:8020/user/hive/warehouse/tpcds_30000_parquet_new.db/catalog_sales/cs_sold_date_sk=2451404/1544efc7887e3b9b-40c41e99000000c1_1344577430_data.2.parq
Error(255): Unknown error 255
Root cause: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
I0627 23:31:50.616274 171066 query-state.cc:337] Cancel: query_id=3b4aeb6665ce8dae:7757b04300000000

Error from coordinator node

W0627 23:32:02.232866 191757 UserGroupInformation.java:1920] PriviledgedActionException as:impala/[e-mail] (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
W0627 23:32:02.237445 191757 Client.java:709] Couldn't setup connection for impala/[e-mail] to vc1512.foo.com/10.:8020
Java exception follows:
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:594)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:396)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:761)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:757)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:756)
	at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1555)
	at org.apache.hadoop.ipc.Client.call(Client.java:1478)
	at org.apache.hadoop.ipc.Client.call(Client.java:1439)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
	at com.sun.proxy.$Proxy19.delete(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:535)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy20.delete(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2062)
	at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:684)
	at org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:680)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:680)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
	... 27 more
W0627 23:32:02.238351 191757 UserGroupInformation.java:1920] PriviledgedActionException as:impala/[e-mail] (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for impala/[e-mail] to vc1512.foo.com/10.:8020
I0627 23:32:02.939185 45977 impala-beeswax-server.cc:233] close(): query_id=3b4aeb6665ce8dae:7757b04300000000
I0627 23:32:02.939621 45977 impala-server.cc:952] UnregisterQuery(): query_id=3b4aeb6665ce8dae:7757b04300000000
I0627 23:32:02.939944 45977 impala-server.cc:1037] Cancel(): query_id=3b4aeb6665ce8dae:7757b04300000000
I0627 23:32:02.971863 45977 query-exec-mgr.cc:138] ReleaseQueryState(): query_id=3b4aeb6665ce8dae:7757b04300000000 refcnt=1
I0627 23:32:03.003536 45977 impala-server.cc:1713] Connection from client 10.17.187.36:48502 closed, closing 1 associated session(s)
I0627 23:32:03.011557 45977 status.cc:123] Session closed
    @           0x90cf99  impala::Status::Status()
    @           0xc4421a  impala::ImpalaServer::CloseSessionInternal()
    @           0xc44900  impala::ImpalaServer::ConnectionEnd()
    @           0xb39640  impala::ThriftServer::ThriftServerEventProcessor::deleteContext()
    @          0x1d24d73  apache::thrift::server::TThreadPoolServer::Task::run()
    @          0x1d0be59  apache::thrift::concurrency::ThreadManager::Worker::run()
    @           0xb3f119  impala::ThriftThread::RunRunnable()
    @           0xb3fb72  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @           0xd75742  impala::Thread::SuperviseThread()
    @           0xd75fe4  boost::detail::thread_data<>::run()
    @          0x102f32a  thread_proxy
    @       0x37dd207aa1  (unknown)
    @       0x37dcee893d  (unknown)
W0627 23:32:12.421926 44973 UserGroupInformation.java:1403] Not attempting to re-login since the last re-login was attempted less than 60 seconds before. Last Login=1498631482824
W0627 23:32:12.422919 44973 UserGroupInformation.java:1027] No TGT after renewal. Aborting renew thread for impala/[e-mail]
I0627 23:43:13.903360 44895 init.cc:294] Successfully renewed kerberos TGT

Issue Links

• duplicates: IMPALA-5672 Queries occasionally fail with GSSException: No valid credentials provided