[IMPALA-64] Queries fail on secure environment due to an impalad's Kerberos ticket expiring - ASF JIRA

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: Impala 0.6
Fix Version/s: Impala 0.6
Component/s: None
Labels:
None

Description

Queries occasionally fail targeting impalad when running against a secure environment. This seems to be because the instance is unable to talk to other impalads in the cluster due to its ticket expiring. These are short running queries (should complete in < 10 seconds)

Tuple(id=0 size=36 slots=[Slot(id=0 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=2 type=INT col=2 offset=8 null=(offset=0 mask=2)), Slot(id=4 type=INT col=4 offset=12 null=(offset=0 mask=4)), Slot(id=6 type=INT col=7 offset=16 null=(offset=0 mask=8)), Slot(id=10 type=INT col=10 offset=20 null=(offset=0 mask=10)), Slot(id=11 type=FLOAT col=12 offset=24 null=(offset=0 mask=20)), Slot(id=12 type=FLOAT col=19 offset=28 null=(offset=0 mask=40)), Slot(id=13 type=FLOAT col=13 offset=32 null=(offset=0 mask=80))])
Tuple(id=1 size=12 slots=[Slot(id=1 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=17 type=INT col=6 offset=8 null=(offset=0 mask=2))])
Tuple(id=2 size=24 slots=[Slot(id=3 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=8 type=STRING col=1 offset=8 null=(offset=0 mask=2))])
Tuple(id=3 size=56 slots=[Slot(id=5 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=14 type=STRING col=1 offset=8 null=(offset=0 mask=2)), Slot(id=15 type=STRING col=2 offset=24 null=(offset=0 mask=4)), Slot(id=16 type=STRING col=3 offset=40 null=(offset=0 mask=8))])
Tuple(id=4 size=24 slots=[Slot(id=7 type=INT col=0 offset=4 null=(offset=0 mask=1)), Slot(id=9 type=STRING col=24 offset=8 null=(offset=0 mask=2))])
Tuple(id=5 size=104 slots=[Slot(id=18 type=STRING col=-1 offset=72 null=(offset=0 mask=10)), Slot(id=19 type=STRING col=-1 offset=88 null=(offset=0 mask=20)), Slot(id=20 type=BIGINT col=-1 offset=8 null=(offset=0 mask=1)), Slot(id=21 type=BIGINT col=-1 offset=16 null=(offset=0 mask=0)), Slot(id=22 type=DOUBLE col=-1 offset=24 null=(offset=0 mask=2)), Slot(id=23 type=BIGINT col=-1 offset=32 null=(offset=0 mask=0)), Slot(id=24 type=DOUBLE col=-1 offset=40 null=(offset=0 mask=4)), Slot(id=25 type=BIGINT col=-1 offset=48 null=(offset=0 mask=0)), Slot(id=26 type=DOUBLE col=-1 offset=56 null=(offset=0 mask=8)), Slot(id=27 type=BIGINT col=-1 offset=64 null=(offset=0 mask=0))])
E0214 10:00:07.121000 24251 authorization.cc:72] Kerberos: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired)
I0214 10:00:07.159731 24251 status.cc:40] Couldn't open transport for 10.20.80.123:22000(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Ticket expired))
    @           0x852262  impala::Status::Status()
    @           0x818315  impala::ThriftClientImpl::Open()
    @           0x7b6284  impala::BackendClientCache::GetClient()
    @           0x9b5cca  impala::DataStreamSender::Channel::Init()
    @           0x9b7cbf  impala::DataStreamSender::Init()
    @           0x7fa8e5  impala::PlanFragmentExecutor::Prepare()
    @           0x68bf4b  impala::ImpalaServer::FragmentExecState::Prepare()
    @           0x69c89f  impala::ImpalaServer::StartPlanFragmentExecution()
    @           0x69d876  impala::ImpalaServer::ExecPlanFragment()
    @           0x858cb0  impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment()
    @           0x852f09  impala::ImpalaInternalServiceProcessor::dispatchCall()
    @           0x69e73b  apache::thrift::TDispatchProcessor::process()
    @          0x12c977a  apache::thrift::server::TThreadedServer::Task::run()
    @          0x12cbe72  apache::thrift::concurrency::PthreadThread::threadMain()
    @     0x7ffd037bf7b6  start_thread
    @     0x7ffd02d829cd  clone

I also have seen cause failures when communicating with the Hive Meta Store Service in secure mode. It appears Impalad is not kinit'ing frequently enough:

13/02/20 11:52:42 INFO hive.metastore: Trying to connect to metastore with URI thrift://impala-centos57-3.ent.cloudera.com:9083
13/02/20 11:52:42 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:KERBEROS) from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
13/02/20 11:52:42 DEBUG transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslClientTransport@8dc1f04
13/02/20 11:52:42 ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:163)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:51)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151)
	at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1114)
	at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1103)
	at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:2206)
	at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:334)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:706)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
	... 40 more

Version: impalad version 0.6 RELEASE (build ac205b9d3c02cbc82f306d68fa7633790fb7a6ad)

Queries fail on secure environment due to an impalad's Kerberos ticket expiring

Details

Description

Attachments

Activity

People

Dates